<<< NOTESD$:[NOTES$LIBRARY]NTA-PROGRAM.NOTE;2 >>> -< nta-program >- ================================================================================ Note 5.0 Possible OpenVMS Product Concepts 2 replies STAR::BMATTHEWS 1863 lines 9-AUG-1995 09:59 -------------------------------------------------------------------------------- *********************Digital Confidential********************** Possible OpenVMS Product Concepts My view of the future for OpenVMS is as the highest performance, most scalable, most available large scale system accessed from PC clients and NT servers. In addition I see OpenVMS being deployed as a black box storage server and application server. The OpenVMS Strategy is a two prong strategy of making OpenVMS a premier Business Critical Server and providing increasingly seamless interoperability with Windows and Windows NT. The second prong of the OpenVMS strategy is known as NT Affinity. Further the NT Affinity strategy contains four themes; Seamless Access to Data, System Management from Windows NT, Application Development, and Windows NT I14Y. The OpenVMS group is beginning the formal planning of the NT Affinity deliverables thru the new PM&D planning processes. The first step of which is the generation and evaluation of product concepts. Following is a list of possible product concepts with a list of possible supporting projects for each concept. Some concepts are really project ideas and need to be raised a level and expanded to be real product concepts. The product concepts are grouped under the NT Affinity "themes". For each theme there will be a team of people from engineering, product management, and marketing that will generate a one page statement of opportunity for each product concept and then determine which product concepts will be fully explored. Following the list of product concepts is some background information on each product concept. Note that these are all possible product concepts and not funded projects. Note that this is raw, rough material that is input to work in progress and should not be viewed as any sort of finished product or plan. Do not forward. *********************Digital Confidential**********************  *********************Digital Confidential********************** Product Concepts for the Theme - Seamless Access to Data -------------------------------------------------------- o Central Storage Server An OpenVMS cluster running the Dollar server as a central storage server for multiple Windows NT systems, possibly in separate security domains to provide a highly available storage, centralized management of storage, consolidation of spare disk storage, and a single point for offline storage management. - Dollar Client on NT to provide a broad market - Dollar Server on OpenVMS to provide the enterprise value of OpenVMS and clusters to the NT clients - Heterogeneous client support to be able to centrally manage NT files on an OpenVMS system - ARGUS Disk mgmt to be able to manage online storage from Windows NT easily and completely - NT front ends to near-line and off-line storage, take the OpenVMS based storage mgmt utilities and provide NT front ends to easily and completely manage the tapes. - Deliver in 18-21 months both a package and the general technologies which will have a broader value. o OpenVMS Integrated LAN Services Integrate the PATHWORKs server with OpenVMS to provide highly available and scalable LAN services as a peer to Windows NT for increasingly seamless interoperability between OpenVMS and Windows NT. - Incorporate and integrate PW server with OpenVMS - Bundle 1st 10 clients with OS? - Single sign-on - NT Registry - Heterogeneous file support - I14Y w/ Bristol - RPC over named pipes o OpenVMS as a Distributed Object Data Server An OpenVMS cluster for highly available storage of object data to Windows and Windows NT clients. - Object Storage on Dollar to support OFS from Cairo - OLE/COM on OpenVMS integrated w/ Bristol WIN32 API - ODI's OODB on OpenVMS - Support DB vendors OO initiatives *********************Digital Confidential**********************  *********************Digital Confidential********************** Product Concepts for the Theme - Windows NT I14Y -------------------------------------------------------- o OpenVMS Server component/extension of MS BackOffice MS BackOffice is a collection of products marketed under the umbrella of MS BackOffice. It is expected that by the end of the calendar year MS will create a package containing all the BackOffice products and sell it at a substantially lower cost than the sum of the pieces. MS BackOffice contains the SQL Server, System Management Server, Mail/Exchange, SNA Server, Windows NT Server, and Windows NT Workstation. The Product concept is to add a component to MS BackOffice to enhance Windows NT ability to connect and interoperate with OpenVMS. The idea is that by having an OpenVMS product as part of BackOffice we would get some Microsoft marketing and would get Microsoft to ship some sw that enhances OpenVMS connection to Windows NT. We would also get a large percentage of the revenue from the sale of the OpenVMS component. A couple possibilities for the OpenVMS component of MS BackOffice The ARGUS PC clients, DECnet for NT, DCE for NT, eXcursion for NT, RDB/SQL dialect, LAT for NT, a "backup gateway" that uses the Arcada and/or Cheyenne frontends to a VMS connected tape subsystem, CMS client for NT, VTX client for NT, ... Now DCE for NT and eXcursion for NT may not be good choices because they don't require OpenVMS servers and we want to be able to charge for those, but clients that require OpenVMS servers like ARGUS, CMS, VTX, Dollar, transports like DECnet and LAT etc. may be good choices. Other ideas? o OpenVMS NT Affinity Out-of-the-Box Server Provide all of the software necessary to operate OpenVMS as a server in an easy to install, configure, license, and operate package. - Bristol runtime - PW Server runtime - Softwindows - Mgmt agents and servers and SNMP runtime and ARGUS server - Dollar server runtime - Middleware runtime, DCE, ObjectBroker, DECMessageQ, TCP/IP o Integrate OpenVMS and Windows NT TP Distributed transactions between OpenVMS and Windows NT will become increasingly important as transactional semantics are added to objects. Integrate OpenVMS's clusterwide 2 phase commit with object servers on OpenVMS that interoperate with Windows NT TP. - Use DECdtm and the XA interface, do we need a TC interface? - Build a FTM to interface to Microsoft TC and possibly Encina - Implement transaction semantics in COM using DECdtm on OpenVMS o OpenVMS/Windows NT Connectivity/Management Description - TBD *********************Digital Confidential**********************  *********************Digital Confidential********************** Product Concepts for the Theme - System Management -------------------------------------------------------- o Windows NT System Management Station for OpenVMS A Windows NT view of OpenVMS for system managers of OpenVMS. A single management staff to easily manage both OpenVMS and WIndows NT systems. Easy to install, configure, and operate Windows NT as a management station for OpenVMS. - Client package for WNT and/or Server package for OpenVMS - Tools to manage OpenVMS from WNT - Some from OpenVMS, LPs, and 3rd parties - Tested, integrated, easily installed, config'd, used - Manageworks, ARGUS, Polycenter xxx - NT front ends to Storage Mgmt - Tools to read OpenVMS mgmt files and generate ACCESS/EXCEL o Enterprise Mgmt of OpenVMS OpenVMS easily managed as a component of a complex enterprise wide network of machines with all the protocols, transports, and agents needed. - Enterprise Mgmt SDK w/ MIB compiler, ARGUS APIs, report templ. - Enterprise Mgmt Server w/ agents, SNMP V2/Host MIB, TCP/IP *********************Digital Confidential**********************  *********************Digital Confidential********************** Product Concepts for the Theme - Application Development -------------------------------------------------------- o 3 tier C/S Application Development Station Application development tools to support OpenVMS as a target for deployment of 2nd and 3rd tier parts of a 3 tier C/S application. - Training program - NT and OpenVMS HW - Software for 3 tier Entera, Forte', DECadmire with trial PAKs - Middleware ADKs for DCE, DECMessageQ, ObjectBroker - Example 3 tier applications - SI offerings o Windows NT Application Development Station for OpenVMS OpenVMS application developers able to use Windows NT and Windows NT development tools to write, modify, and debug OpenVMS applications remotely to make OpenVMS application development more attractive to programmers. - Remote RPC access to OpenVMS APIs - OpenVMS APIs native on Windows NT - Cross platform development tools from Windows NT for OpenVMS o Common code for WIndows NT and OpenVMS Applications can be coded to one API for Windows NT and deployed on both OpenVMS and Windows NT. - WIN32 API from Bristol - OLE/COM API - Common cluster APIs o OpenVMS as a Distributed Object Application Server OpenVMS as a highly available and scalable object server with tight integration to the desktop thru distributed OLE/COM. - ObjectBroker integration with OpenVMS and DCE - OLE/COM Integration - OpenVMS COM clusteraware server objects - WIN32 API for portable COM server objects - Wrapping existing applications as objects - Transactional semantics for objects - XA interface to DECdtm, do we need a TC interface - FTM gateway to Microsoft TC, perhaps also Encina *********************Digital Confidential**********************  *********************Digital Confidential********************** Product Concepts for the Theme - Business Critcal Server -------------------------------------------------------- o Integrated OpenVMS and Windows NT Clusters. - Common APIs for cluster aware applications to make it easy to write applications that scale and are highly available in a cluster, both OpenVMS and WIndows NT cluster. - High speed communication path between OpenVMS and Windows NT for efficient and effective client/server computing, for example OpenVMS and WIndows NT systems on the same Memory Channel Hub. - Cooperation of OpenVMS and WIndows NT clusters. For example, allowing a Windows NT Management station to hold a quorum vote in an OpenVMS cluster. o OpenVMS as a highly available application server of choice - Combination of underlying routines and class libraries - Transport independent cluster alias? - Upgrade IPC for communications and connection to non-IPC code - Enhance RPC performance - TxRPC that works with Microsoft WNT & DECdtm - Encapsulate recoverability attributes in class libraries - Transaction recovery manager - Per thread security - Monitoring/debug/performance/test tools for C/S applications - Tools to help partition application for scaling - Load balancing API and tools - Failover/Failback Services - Cluster "doorbell service" that arbitrates to standby servers - Build an example highly available server application *********************Digital Confidential**********************  *********************Digital Confidential********************** Product Concepts for the Theme - Dollar Futures ----------------------------------------------- o File system and backup via CD Jukeboxes and Dollar - Jukeboxes of write once CD ROMs possibly video size CD ROM - Exploit the combination of CDR technology becoming a consumer electronic item and Dollar's LFS technology for read/write access to a large write-once storage. - Backup media for unattended backup and a step towards self managed storage o Data vaulting via continuous backup/restore of Dollar - Exploit the dollar LFS technology to provide data vaulting via continuous online backup and restore. Combined with NT client could provide data vaulting for NT. Perhaps local non-LFS data could be "mirrored" as LFS. o License the Dollar technology - License the Dollar technology to other companies to make money - Push Dollar C/S protocol as a future storage/SCSI standard o Continuous backup/recovery - Explore opportunities for continuous online backup applications o Hybrid in place file system and log-structured file system to support database in place writes and log-structured online backup - Provide online backup for databases and in place writes - Support non-sophisticated databases by doing write in place until a snapshot is requested, then begin logging writes, replay the log after the backup is complete and resume in place writes. o Heterogeneous file support - Explore possible business opportunities that exist with heterogeneous file support - With a Dollar NT server explore business opportunities of a common ondisk structure across operating systems o High performance data servers based on Dollar - NFS Server on Dollar - Server application to access w/o going thru F64 o OpenVMS clients outside the cluster - Client Dollar software on VAX ala NT to provide file access to OpenVMS nodes outside the cluster o VAX access to Dollar - VAX client access to Dollar to allow VAX to Alpha migration of data for applications that want to stay on VAX o Improved HSM and Dollar by shelving Dollar segments Couple shelving with Dollar such that once a volume gets X% full then shelving kicks in to copy files to offline/nearline storage and to extend the life of the log and delay the device full condition. Shelve segments of files instead of all the file for large files with some seldomly referenced segments. o Storage management via the Dollar cleaner Compression on the fly possibly by the cleaner for "cold" data. Defrag via the cleaner, especially of hot files. *********************Digital Confidential**********************  *********************Digital Confidential********************** Clusters: Integrated OpenVMS and Windows NT Clusters. - Common APIs for cluster aware applications to make it easy to write applications that scale and are highly available in a cluster, both OpenVMS and WIndows NT cluster. - High speed communication path between OpenVMS and Windows NT for efficient and effective client/server computing, for example OpenVMS and WIndows NT systems on the same Memory Channel Hub. - Cooperation of OpenVMS and WIndows NT clusters. For example, allowing a Windows NT Management station to hold a quorum vote in an OpenVMS cluster. Current engineering investigations details below: NT Affinity & Clusters Investigation Tasks 17 July, 1995 Greg Jordan and Jim Krieger 1. Understand the Client/Server Market Place and NT The Client/Server market place is a new focus for current team members. It is important that we obtain a feel for this market place to help us investigate solutions. In looking at the Client/Server Market place, our focus will be more on middleware and back end servers (2nd and 3rd tier systems) since this is where Clusters would exist in a Client/Server environment. o Understand Middleware and Back End Servers Obtain an understanding of the types of middleware and back end packages used to implement Client/Server applications. o Understand NT Clusters We need an understanding of NT Clusters. This will include a detailed comparison between VMSclusters and NT Clusters, what is the same, what is different, what cluster aware APIs are on each platform. o Understand Who Our Customers Are A major re-occurring theme at the Database & Client/Server World was application independence. The applications shouldn't care what back-end database is used. They shouldn't care what front-end display mechanism is used. They should be plug and play. The business logic should not have any system specific code. A single source should run on numerous platforms. For the above model, our customers will not be application developers, but rather middleware and back end developers. We need these products to take advantage of Cluster technology such that the application gains the benefits by coding to the standard middleware or back end product APIs. There also exists some amount of in-house development which is done on VMS for deployment on VMS. Are the needs of these customers the same as the needs of the middleware and back end developers? 2. Investigate the Cluster Capabilities Our Customers are Missing Developing complex cluster aware applications has never been an easy task. There a numerous issues which must be addressed and solved. o Investigate Missing Infrastructure There are a number of capabilities missing which make it more difficult to produce complex cluster aware applications. Some examples of missing capabilities that customers have been asking for include cluster wide mailboxes, cluster wide global sections, and inter cluster process-process communication. Determine what cluster programming capabilities are missing from VMSclusters and their relative importance. As part of this project, we plan to solicit needs from internal middleware groups, similar to what we are currently doing with with Pathworks. o Investigate APIs for Cluster Functions To produce a cluster aware application, there are a number of problems which most applications need to solve. These include failover, load balancing, clusterwide caching, etc... Can we provide general purpose APIs to solve the above problems such that it become very easy to write a high performance, high availability cluster aware application? 3. Investigate NT Affinity & Clusters This area of the investigation will address issues between VMSclusters, NT, and NT Clusters. There are a number of questions we will try and answer in this part of the investigation. A. What are the primary migration and development goals? o Should NT Cluster APIs be available on VMS? o Should VMScluster APIs be available on NT? o Should any newly developed cluster aware functional APIs be provided on both NT and VMS? o Should any new functions be provided as objects in class libraries or as callable functions or both? B. Should we investigate allowing an NT node to provide a vote into a VMSclusters? This BRS request would allow the OMS to hold the quorum vote for a BRS site. This is NOT a request for VMS and NT to exist in the same cluster. C. Investigate the need for both VMSclusters and NT Clusters to exist on the same Memory Channel Hub. Such a configuration could allow high speed communications between VMS and NT. *********************Digital Confidential**********************  *********************Digital Confidential********************** Objects: OpenVMS as a Distributed Object Data Server - Object Storage on Dollar to support OFS from Cairo - OLE/COM on OpenVMS integrated w/ Bristol WIN32 API - ODI's OODB on OpenVMS - Support DB vendors OO initiatives OpenVMS as a Distributed Object Application Server - ObjectBroker integration with OpenVMS and DCE - OLE/COM Integration - OpenVMS COM clusteraware server objects - WIN32 API for portable COM server objects - Wrapping existing applications as objects - Transactional semantics for objects - XA interface to DECdtm, do we need a TC interface - FTM gateway to Microsoft TC, perhaps also Encina Details below from Mark Sorenson OpenVMS's Enterprise OO Strategy MES012795 1. Executive Summary: Object-Oriented (OO) technologies are becoming increasingly important to software developers promising such benefits as increased software re-use, rapid application development (RAD) of distributed applications, and easy integration of new applications with legacy software and data. The OpenVMS OO strategy seeks to exploit OpenVMS's leadership capabilities of our highly available VAXcluster systems, state-of-the art file storage/archiving and multivendor interoperability capabilities, and the overall reliability of the feature rich OpenVMS Operating System running on high performance Alpha systems. Complementing the OpenVMS family of products and systems is a portfolio of leadership Enterprise Objects Software products enabling OpenVMS enterprise wide server systems to easily access and provide secure services to Microsoft OLE/COM based departmental (WNT) server and desktop applications. The OpenVMS OO strategy manifests itself in two primary initiatives: - The OpenVMS Object Data Server, providing enterprise-wide object storage capabilities. - The OpenVMS OO Distributed Object Application Server In support of these two strategic initiatives are two key programs which enable and support OpenVMS's OO Strategy: - The Enterprise Objects Software Product Portfolio - The OpenVMS Mission Critical Object Services 2. Initiative 1: The OpenVMS Object Data Server: This strategy focus on providing corporate wide data to WNT departmental servers and PC clients. The two flavors of data servers include an Object database server and an object file server. The Object database server can store corporate data in a pure OODB such as ODI's ObjectStore, as objects in traditional relational databases, via RDB or ODBC interfaces, via RMS, or in the emerging OO API's RDB vendors are building into their products -- Sybases 'Build Momentum' or the OO API scheduled to be delivered in Oracle8. The OpenVMS object file server, based on the Common Object Server (COM), serves corporate files to OLE/COM based applications providing security, availability, and high-performance storage. The OpenVMS Object Data Server Customer Profile: - May be encapsulating legacy data to distribute to departmental and desktop systems. - Seeks simplified access to corporate data via OO interfaces. Enables rapid application development and deployment to adapt to changing business needs. - Requires highly available, reliable systems which can provide security and intelligence to corporate data resource access. - Has large data storage requirements, e.g. 10's of GB's to 10's of TB's Competitive Strengths: The primary competitive strength of OpenVMS Object Data Server is the exploitation of OpenVMS's traditional availability, reliability, and scalability attributes coupled with the state-of-the art Dollar file system in an OO data server. The supporting family of OpenVMS products which enable many of our attributes - VAXclusters, Shadowing, Journaling, etc... are well know. The capabilities of the dollar log-structured file system are less well known but are key to the Object Data Servers success, specifically; - Write performance up to 10 times faster than today's existing OpenVMS AXP Files-11 system - High storage capacity -- up to 10 Tbytes of data online - Highspeed, continuous backup with true incremental backup which only saves data written since the last backup rather than the entire contents of all of the files updated since the last backup. - A fully-compatible RMS file system and backup interface enables existing OpenVMS AXP applications to take advantage of log-structured technology without modification. Future considerations include; - An NTFS and/or OFS interface to the dollar file system - Multi-threaded object servers to exploit SMP scaling - Object Server failover capabilities - Object Server load balancing - 64 bit support 3. Initiative 2: The OpenVMS Distributed Object Application Server: The Distributed Object Application Server initiative allows MIS departments to encapsulate legacy applications or to build new mission-critical applications as clusterwide corporate objects that can be accessed by PC and WNT clients using OLE and COM. Departmental/local PC or WNT based applications can then be easily and rapidly built which can access the corporate objects thru well known OO api's. Optimally the object server should be "close" to the data to take advantages of the object server attributes noted above, and to minimize network bandwidth needs for performance reasons. The OpenVMS Distributed Object Application Server Customer Profile. - OpenVMS installed base legacy applications seeking to use OO technology to extend the life of their applications and to move to a more flexible client-server architecture. - MIS departments using OpenVMS and OO technologies to right-size their existing mainframe mission critical application - New applications builders (MIS, ISV's) who require OpenVMS's mission-critical attributes for the server side of WNT OLE/COM applications. - OLE/COM applications which have server side compute intensive requirements satisfied by OpenVMS Alpha systems, and/or require the object storage capabilities noted in initiative 1. Competitive Strengths: - The leadership OO infrastructure of our EOS product set including ObjectBroker/COM, DCE, Pegasus, Forte, and DECmessageQ. - Unmatched price/performance instantiated by Alpha systems and supported by advanced technologies such as the dollar file system, 64 bits and kernel threads. - The proven, reliable OpenVMS Operating system with it's high availability, C2 security, and ease of manageability. Futures Include; - Tighter network wide security and directory services integration - Migration from OLE portal support to full COM support - Tools providing powerful class structures, base class libraries, and cross-language sharing on COM - Load balancing and failover capabilities 4. The Enterprise Objects Software Product Portfolio Need someone from EOS to supply a brief overview of the EOS product strategy, and short descriptions of each product. I suspect that these products will be covered in the application development report, but I believe this report would be incomplete without a reasonable overview. I presume this should be a quick cut-and-paste from an existing EOS strategy document. Mark Bramhall or Mark Storm - could you provide? 5. The OpenVMS Mission Critical Object Services It is a primary goal to make OpenVMS's key attributes available to OO data servers and OO Application servers with little or no additional coding to applications. Simply running applications on OpenVMS AXP VAXclusters should provide substantial benefits to mission-critical data servers and applications. However there may be key OpenVMS services that if encapsulated and provided an OO interface that may be of substantial value. This advanced development project will provide OO interfaces to selected OpenVMS services. Services under consideration include: - a high performance backup object, allowing data to be backed up continuously and remotely. - a DECdtm object, which provides 2 phase commit capabilities - a batch/print object, allowing ease of scheduling, checkpoint/restart - Resource Brokering/load balancing and failover object - Management objects (storage/clusters/users) permitting the export of OpenVMS management objects via COM interfaces and allowing seamless integration into other management directors 6. Windows NT Affinity The design center for the OpenVMS Object strategy is in it's simplest form: PC's on the desktop, Windows NT departmental servers, and OpenVMS VAXclusters as the corporate data and application server. Although interoperability with systems outside this model is an important requirement, the priority is the Windows-WNT-OVMS model. For example, although CORBA compliance is a requirement and desirable, the strategic focus is OLE/COM and is where most of the work should be focused. In order to make WNT departmental servers work well with OpenVMS enterprise servers, more seamless integration with WNT services is required. Some of these projects could be integrated with OpenVMS, others in a separate toolkit. Further detailed investigation is required but some possible opportunities include: - Security integration including single signon, authorization, and authentication. Automated mapping of ACL's to ACE's, UAS to SYSUAF to DCE Security. Provide the ability to import and export security attributes (users, p/w...) - Directory Services integration including WNT Object Directory Object, NSID, CDS/GDA. - Interoperability between the OpenVMS and WNT event loggers - Ensure relevant OpenVMS services are remotely manageable and utilize RPC - Enable OpenVMS services to be controlled by the Service Control Manager for remote start/stop/pause/continue operations. - Provide hooks so that OpenVMS development tools can integrate with WNT tools such as: WIN32 API Profiler, Performance Monitor, Working Set Tuner, and Call Attributed Profiler. - Provide the best compatibility between Microsoft RPC and DCE RPC including named pipes support, and IDL - MIDL compatibility. - UNICODE support (?) WinSocket support (?) NOTES TO REVIEWERS: Peter L - suggestions and review please! Can you articular the GUI aspect you spoke of the other day? Bill M - help me with the Backoffice stuff? 7. What We Have Today, 6-12 months, and in the Future: Today: - ObjectBroker, with OLE portal. Supported on OVMS, WNT, Windows and 11 additional platforms - DCE base services, Supported by Digital on OVMS, WNT, Win, and OSF/1 - DECmessageQ - Forte - ObjectFlow - Argus V1 - Dollar Directed Developers Kit - ODI 6-12 Months: - ObjectBroker, with full COM support, support of DCE Security & Naming - Pegasus Development Tools - DCE, integrated login, GSSAPI. Enhanced Microsoft RPC and IDL compatibility - Resource Broker - Dollar - K-Threads - 64 bits - Argus Vnext.next - WNT/OpenVMS Server Development Toolkit SDK Future: - OpenVMS Mission Critical Object Services - Argus Management Objects - OpenVMS Mission Critical Objects - Dollar integration with the Cairo OFS - Dollar support of NTFS - WNT/OpenVMS Server Development Toolkit V1.0 - Encina/CICS support (??) - Black-box OpenVMS Object Servers 8. Impact on Attracting Applications: MES 9. Issues: - For DB vendors to realize the benefits of dollar, some collaboration is required. Are they motivated and interested? - RDB's typically have their own backup utilities, how much does that decrease dollars value. - Is it technically possible to extend dollar to NTFS? Cairo OFS? - Is our development environment adequate? Can we develop a model that maximizes WNT and WNT tools as the development platform? - Encina availability on OpenVMS? Object needs as seen by Mark Bramhall The current strategy's "shoring up CORBA leadership while monitoring OLE/COM" in FY96/97 unfortunately will not work. We must be aggressively investing in OLE now or the opportunities will pass us by. Tighter integration with Digital UNIX, OpenVMS, and Windows NT all require either a system-level object model or tight PC connectivity or both. The current CORBA-only code base cannot serve as a system-level object model while OLE/COM can and OLE/COM is obviously much better for tight PC interoperability. Opportunities to partner with the Win32 environment providers (Bristol and MainSoft) for all UNIXes as well as OpenVMS depend on OLE/COM. Such a partnership would provide all of Digital's OS's with a strong OO story, PC integration story, and ISV environment story. Well over three quarters of our potential technology partner talks center around OLE/COM. It is what Open Environment wants; it is the system-level model Oracle needs; it is the PC connectivity Informix requires; etc. We must publicly state a firm direction towards an OLE-based infrastructure with enough CORBA compliance and interoperation to support our current customers as well as new customers. Part of the direction must be the importance of supporting development tools and environments, deployment tools, management tools, etc. This does not mean we should drop our current customers (who are by definition using a CORBA-only product). The QAR backlog for the released ObjectBroker V2.5 must be tackled to prevent any current customer exposure liability. ObjectBroker V2.6 must be released with its CORBA enhancements, security, and a better desktop connection. It does mean we should understand the longevity of the current code base and make plans now to ensure moving to a single code base as soon as possible. The first important commitment is to ensure we complete the E1 (first complete milestone of "Emily", the OLE-based work project's code name) as planned and scheduled. This means ensuring the the resources originally committed to it stay committed and the attrition vacancies (most notably the project manager!) filled ASAP. E1 not only kick-starts the OLE-based technology path, it also had the seeds for many of the needed new thrusts: better desktop connection, system-level object model, development environment and tools, etc. Furthermore, while we probably will not generally customer-ship E1, it will yield rewards in splashy Object World demos (mid-August timeframe) and outbound technology for potential partners like Open Environment, Bristol, MainSoft, etc. The next needed commitment is to find (from outside of the current resources) the resources needed to get the ObjectBroker V2.5 QAR backlog under control and to ensure a timely ObjectBroker V2.6 release. ObjectBroker V2.6 is a more production-oriented release with CORBA enhancements, e.g., C++ bindings, and DCE security. These additional resources can also bring over the Windows desktop work E1 is doing, speed up the OLE/CORBA bridge work to glue it to V2.6, and thus enhance V2.6's desktop connection. Finally, some of these resources should be put to the task of setting up the general source control, build, and test set of systems needed for fast-paced development of multi-platform software like our object products. This work is immediately needed by ObjectBroker V2.5/V2.6 but should be set up as the development environment supporting "Emily" and, more importantly, the follow on single merged project that follows ObjectBroker V2.6 and "Emily." The number of resources needed is non-trivial and it is important that the full management chain be engaged in acquiring them. A major help is the clear and crisp statement of direction and obviously full management support behind same. Initial use of temporary help and assistance in finding new permanent hires is also needed. We have market momentum, albeit declining. Internally our engineering morale needs bolstering. This is a critical juncture. With the right clear, crisp, and public statement of direction and application of resources, we can have the needed momentum to let us "hit the ground running" for FY96 and beyond. This is the momentum needed to win. *********************Digital Confidential**********************  *********************Digital Confidential********************** TP: Integrate OpenVMS and Windows NT TP - Use DECdtm and the XA interface, do we need a TC interface? - Build a FTM to interface to Microsoft TC and possibly Encina - Implement transaction semantics in COM using DECdtm on OpenVMS Details below from Jim Johnson: From: STAR::MOVIES::JJOHNSON "James Johnson, M/S: EDO-13, DTN: 824-3407" 28-JUN-1995 11:27:34.14 To: CHANDLEY,PLAYFORD CC: DOHERTY,STAR::JFALLON,STAR::BMATTHEWS,STAR::KALER,JJOHNSON Subj: some thoughts on what we could do w/ tx commitment & NT aff. Subject: Some opportunities for DECdtm As the NT affinity direction has started to gather steam, I've found myself repeatedly thinking about DECdtm. It seems to me that if we are claiming to be the enterprise backbone server we need to provide the features for reliable distributed computing. What I've not been too sure of until recently is what direction we should take with DECdtm in order to provide the best fitting services for NT affinity. I think that Microsoft's recent announcements around their transaction coordinator has made the direction obvious. We should provide foreign transaction manager (FTM) gateways to Microsoft's TC protocol, and to Encina's protocol. We should enhance the DECdtm state machine, if necessary, for these gateways to work reasonably. We should also look to provide an API layer to the DECdtm services that is TC compatible. We should work with COM to see that they use these services for their own transaction management (since TC is also hooked to OLE on Microsoft's systems). Finally, we should publish three interfaces: The TC compatible one, the X/Open compatible one, and the proprietary one. The last one should include a section describing how to write an FTM. With these items done OpenVMS would have the infrastructure to support reliable distributed applications, the ability for those applications to include NT components and Encina or X/Open components, and the ability for 3rd parties to extend the interoperability of DECdtm to include other database and TP commitment protocols. Jim. *********************Digital Confidential**********************  *********************Digital Confidential********************** Dollar: My view is to just pick one product concept for Dollar and that the chosen product concept should support the OpenVMS/Windows NT Strategy but there are many opportunities for the Dollar technology. The ones I am aware of are listed below following my favorite product concept for Dollar as a storage server for NT. Central Storage Server - Dollar Client on NT to provide a broad market - Dollar Server on OpenVMS to provide the enterprise value of OpenVMS and clusters to the NT clients - Heterogeneous client support to be able to centrally manage NT files on an OpenVMS system - ARGUS Disk mgmt to be able to manage online storage from Windows NT easily and completely - NT front ends to near-line and off-line storage, take the OpenVMS based storage mgmt utilities and provide NT front ends to easily and completely manage the tapes. - Deliver in 18-21 months both a package and the general technologies which will have a broader value. o Dollar - What are the opportunities for the DOllar technology? What are the pitfalls with the technology and implementation? What will it take to address the opportunities and pitfalls? - Storage Server - OpenVMS, NT, possibly UNIX - Bring the value of OpenVMS Clusters and Dollar to NT - Provide Central storage for multiple systems and LANs to reduce and simplify storage mgmt - CDR - Jukeboxes of write once CD ROMs possibly video size CD ROM - Exploit the combination of CDR technology becoming a consumer electronic item and Dollar's LFS technology for read/write access to a large write-once storage. - Backup media for unattended backup and a step towards self managed storage - Data vaulting - Exploit the dollar LFS technology to provide data vaulting via continuous online backup and restore. Combined with NT client could provide data vaulting for NT. Perhaps local non-LFS data could be "mirrored" as LFS. - License the technology - License the Dollar technology to other companies to make money - Push Dollar C/S protocol as a future storage/SCSI standard - Continuous backup/recovery - Explore opportunities for continuous online backup applications - Accomadating databases in the C/S model, alternate storage server model that isn't always log-structured. - Provide online backup for databases and in place writes - Support non-sophisticated databases by doing write in place until a snapshot is requested, then begin logging writes, replay the log after the backup is complete and resume in place writes. - Heterogeneous support - Explore possible business opportunities that exist with heterogeneous file support - With a Dollar NT server explore business opportunities of a common ondisk structure across operating systems - Servers that access dollar directly for performance - NFS Server on Dollar - Server application to access w/o going thru F64 - OpenVMS clients outside the cluster - Client Dollar software on VAX ala NT to provide file access to OpenVMS nodes outside the cluster - VAX access - VAX client access to Dollar to allow VAX to Alpha migration of data for applications that want to stay on VAX - Improved HSM and Dollar by shelving Dollar segments Couple shelving with Dollar such that once a volume gets X% full then shelving kicks in to copy files to offline/nearline storage and to extend the life of the log and delay the device full condition. Shelve segments of files instead of all the file for large files with some seldomly referenced segments. - Storage management via the Dollar cleaner Compression on the fly possibly by the cleaner for "cold" data. Defrag via the cleaner, especially of hot files. *********************Digital Confidential**********************  *********************Digital Confidential********************** MS BackOffice: MS BackOffice is a collection of products marketed under the umbrella of MS BackOffice. It is expected that by the end of the calendar year MS will create a package containing all the BackOffice products and sell it at a substantially lower cost than the sum of the pieces. MS BackOffice contains the SQL Server, System Management Server, Mail/Exchange, SNA Server, Windows NT Server, and Windows NT Workstation. The Product concept is to add a component to MS BackOffice to enhance Windows NT ability to connect and interoperate with OpenVMS. The idea is that by having an OpenVMS product as part of BackOffice we would get some Microsoft marketing and would get Microsoft to ship some sw that enhances OpenVMS connection to Windows NT. We would also get a large percentage of the revenue from the sale of the OpenVMS component. A couple possibilities for the OpenVMS component of MS BackOffice The ARGUS PC clients, DECnet for NT, DCE for NT, eXcursion for NT, RDB/SQL dialect, LAT for NT, a "backup gateway" that uses the Arcada and/or Cheyenne frontends to a VMS connected tape subsystem, CMS client for NT, VTX client for NT, ... Now DCE for NT and eXcursion for NT may not be good choices because they don't require OpenVMS servers and we want to be able to charge for those, but clients that require OpenVMS servers like ARGUS, CMS, VTX, Dollar, transports like DECnet and LAT etc. may be good choices. Other ideas? *********************Digital Confidential**********************  *********************Digital Confidential********************** OpenVMS as a server of choice: Below is a long list of capabilities that could be added to OpenVMS to make it a premium choice as an application server. From: STAR::EVMS::MOVIES::JJOHNSON "James Johnson, M/S: EDO-13, DTN: 824-3407" 24-APR-1995 04:12:35.01 To: EVMS::STAR::OPNDCE::BMATTHEWS CC: JJOHNSON Subj: RE: My first pass reactions to Jim's server application memo. Steve Jenkins is there an opportunity for some joint work in this area? Hi Bill, I've put my answers in preceded by >>>. Jim. ----------------------------- My quick first pass comments preceded by Bill> Bill, Here's the finished writeup. I hope you find it useful (even if it is fairly long). There are a couple of things not on this list, since they don't have anything to do with bus. critical servers, that I expect to send along as well. Dan, I copied you on this since you were asking for any lists I've got. This is a pretty good list of areas and/or projects that I think we ought to look at (aside from file system things that we've already talked about). Jim. -------------- Subject: Requirements for building a business critical server application The following mail first describes a desirable format for a highly available server, followed by a number of sections that lay out the various needs such a format implies (and where OpenVMS may need some work...). Note that I've assumed a problem space here. If the OpenVMS server market consists of a small number of pre-existing server applications, probably the best thing OpenVMS engineering could do would be to work closely with the providers of these applications. If, on the other hand, the expectation is that there are a number of as yet unwritten server applications that will be deployed on OpenVMS, then we should look at what sort of environment we should provide for them. This mail assumes this latter case. 1. Description of the Server Application In thinking how I'd want to build a highly available server, I came up with the following sort of structure: At the most basic, the structure would use a simple request-response protocol. The server would be multi-threaded, with concurrent requests executing simultaneously in different threads. Each thread would execute under different security profiles, so that the requests would directly impersonate the identity of the requestor. Also, there would be some form of resource control that limits the number of requests simultaneously in progress. Bill> I agree. Wes happens to be a big fan of queueing and doesn't like request/response. I believe request/response when designed well can give reasonable performance. Finally, the requests are typically constructed so that they may either be rolled back, or are idempotent (and may simply be represented). This is then used in concert with a set of deadman locks that provide primary server selection for both startup and failover cases. I've never found it necessary to provide a fault-tolerant server. Instead, I've built highly available servers, with hot standbys, and had the client (typically the lower layer of the client) automatically reconnect to a new server and represent any outstanding work. If the recovery semantics are sufficiently clean this can be done without ever informing the upper layers of the client (or, obviously, the user). To achieve sufficiently clean semantics, I've always found myself either using a transactional mechanism or a careful update policy. While the latter is more traditional in OpenVMS engineering, it has serious maintainability problems -- ones that are not present in the transactional model. Bill> I agree, the above paragraph will hopefully get people thinking about how to write highly available servers. The above paragraphs essentially describes the structure of the Argus server, the TP server, and undoubtedly several more. The difficulties with this level of support involve the amount of work specific to the server infrastructure in comparison to the amount of work that was specific to the server application's mission. Therefore, an efficient and ubiqitous generic server infrastructure would greatly ease the difficulties in building server applications on OpenVMS. The remaining sections list a number of infrastructure areas that could or should be improved. Bill> Who/what group would you suggest be responsible for building this generic server? Can we get this defined and fill in the underlying bits later? Can this generic server be defined in such a way that we can re-do the underlying technology without changing the application that is built on/with the generic server? >>> I'd viewed this as a combination of underlying routines serving an overall >>> suite of class libraries. The class library interface is the one that is >>> at all important to the application developer, thus leaving a lot of >>> latitude for portability in the implementation and support routines. >>> >>> Furthermore, if we first sketch out the shape of the class libraries, such >>> as what is layered on what, we should be able to phase the class >>> interfaces in over some number of releases (this is similar to what has >>> happened with MFC). >>> >>> Because of these views, I'd have said that the group working on the class >>> library should also be responsible for the generic server infrastructure >>> either directly responsible for implementation (preferable), or for >>> coordinating requirements on other groups (where necessary). I'd also >>> suggest that this should be a dedicated team, somewhat separate from the >>> "base" OS functions. This last is in hope that it would encourage >>> thinking about the project as a "layered product", which is what it will >>> be on other platforms. 2. Communication Needs The first area to consider is that of simple communication. This area covers the needs around connectivity, including locating a server, the programming model we'd want to support, and distributed security needs. 2.1 Basic Connectivity Of course, the first problem that a server application must face is that of simple connectivity between the server and all likely client systems. In today's world this must include TCP/IP. However, it should also include a communications layer provides a tranport-independent interface, and offers very high performance, especially for the local case (this is due to the high percentage of RPCs that end up being for other address spaces on the one node). It follows directly on from this that you'd need a transport independent name, and preferably one that was cluster-node transparent. These two attributes both greatly simplify the programming interface and the failover/failback handling. Bill> Today the naming can also be handled by a name server so the client need not know even which cluster the service is on. Binds and re-binds are done by requesting the location of the service of interest to the client from the name server. So I would say that a cluster alias that was transport independent is nice to have but if a name server is needed for other reasons then the name server can provide this function. >>> Excellent. The basic need is to be able to give a service a single >>> name, and that such a name should be usable regardless of underlying >>> transport, and should be usable at some later time (assuming that the >>> service has not logically moved or changed). A name service probably >>> does provide these features. Great! It must also be possible for a server to handle several thousand clients simultaneously. Therefore, the communications mechanism either needs to be connectionless, or support thousands of simultaneously active connections. Bill> DCE RPC addresses this problem by supporting both connectionless and connection oriented, however connections that aren't used for 30 seconds are torned down automatically by DCE RPC and then re=created transparently when the next RPC call is made in order to throttle down the number of active connections. But I agree that scaling into the thousands of connections needs to be designed in upfront. I'd suggest that OpenVMS work on the following areas (some of which are under development already): - IPC. Upgrade it to support TCP/IP, work on performance and stability. Provide a no-additional-protocol mode to allow IPC to connect to non-IPC based services. Bill> I would like to understand what the non OpenVMS client would use to an OpenVMS server that communicated via IPC. I could see RPC or XTI using IPC for local and intra-cluster communication and TCP/IP off cluster. I don't see the gain of IPC on top of TCP/IP vs straight TCP/IP unless the app coded straight to IPC instead of XTI or RPC. >>> This one, and the following few, are probably confusing because there >>> are a number of different ways of getting the attributes around speed, >>> transport independence, and excellent RPC support. Based on your >>> comments, I believe we simply picked different options. Not actually a >>> big deal -- these were just suggestions, and I'm happy to take any of the >>> various alternatives that address the needed attributes. - XTILIB. This library must be made threadsafe. Bill> I agree that to be a possible server API it must be threadsafe. - TCP/IP. Integrate support for TCP/IP as an equal partner to DECnet, work on its performance, integrate with IPC. Bill> I see TCP/IP as under IPC so I don't see a need for TCP/IP to be IPC aware. - DCE RPC. Aggressively pursue improved performance. Do whatever it takes to compete effectively against a procedure call in the local case. Layer on IPC for transport independence and raw intra-cluster performance. Full interoperation with Microsoft's RPC. Bill> I am not sure it is possible to get the performance that you are hoping for out of DCE RPC. On EV4's a small RPC is about 1.7milliseconds, and an ethernet can be driven full bandwidth. >>> The issues we need to think about is how "big" a service needs to be >>> before I should consider offering it as a distributed service. The >>> "bigger" the minimum size is, the less there is pressure on the RPC >>> performance, and the less that we could expect that RPC would be the >>> paradigm for constructing applications out of building blocks, and the >>> more that offering a distributed service would be an exceptional act. >>> >>> One of the biggest contributors to establishing the effective minimum >>> size of an application component to access through RPC is the performance >>> in the local case, and next is the intra-cluster case. While my ideal >>> would be a level of performance that supported RPC as the access >>> mechanism for all but the smallest routines, I'd accept a minimum that >>> supports the performance necessary for composition at the level we've >>> seen in ACMS. We certainly have to offer sufficient performance and >>> capability so that we would stop special-casing targets, like I've been >>> told we do in MAIL (where delivery mechanisms are different if the target >>> is local, is in the same cluster, or isn't either). >>> >>> So maybe 1.7ms local is enough -- depending on our goals around server >>> structures that consist of "composable" units accessed via RPC. - Name Service. Support transport independent naming, with a name that is persistant and globally correct. Provide a single name to access a cluster-wide service, regardless of cluster node (this is to provide a modicum of single-system-image behavior for highly available servers). Integrate with the RPC service. Bill> This is possible. With DCE V1.3 the RPC NSI(Name Service Interface) was augmented with Resource Broker to provide the ability to automatically select the "best" server vs. a random server. 2.2 Server Model While there are a number of possible server models, the ones I'd focus on are variations on request-response. At the simplest, this is just an application directly layered on a communications subsystem. However, the more of this that can be hidden the easier the application portion is to write and maintain. So, I'd certainly use an RPC layer, but that layer must perform in its own right, and needs to be layered on a communications subsystem that performs. Now, for simplicity of failover I'd use a transactional mechanism, so I'd probably want an RPC subsystem that handled transaction propagation (e.g. txRPC). This allows me to safely compose simpler server applications into a larger service without compromising the failure characteristics. Bill> Today's txRPC is implemented as a transport for DCE RPC but txRPC uses the OSI TP stack, so it doesn't require DECnet necessarily on the remote system it does use and require OSI. This was done as part of the MIA work for ACMSxp and folded back into DCE 1.3. >>> Well, blecch. OSI TP is very heavyweight, and isn't very well matched >>> to RPC. I don't know anymore who else intends to support it, but I have >>> my doubts that OSI TP will be the protocol to have in the NT-VMS space. >>> I could well be wrong, and my low opinion of the OSI TP protocol may well >>> be getting in the way. However, I do have my doubts that this will prove >>> to be the right 2PC protocol for this space, and I've equally got doubts >>> that we really yet know which one will be. I suspect that Microsoft will >>> end up releasing their own, probably as they continue to evolve the OLE >>> services. >>> >>> This may be an area where working with Microsoft on what a transactional >>> RPC should look like, and what the protocol should be. If we assume that >>> they are going to provide something in this space, even if buried in the >>> OLE runtime, it would be in our best interest to interoperate. I'd also expect to work in C++, so I'd want an RPC that understood the C++ name mangling, and could hide the communications in an OO manner. This is most likely to be OLE/COM/CORBA based. I'd personally list this as a near term future, rather than a full need today. Bill> Support for C++ was added to the Interface Definition Language (IDL) for DCE V1.3. Not sure what support there is for MS RPC. I would make sure the generic server could have it's comm layer replaced over time. On the OO front, though, I'd look for a common structure to server apps, and codify that commonality in server application framework classes in a manner similar to MFC for Windows apps. This is probably more urgent than a full fledged OO RPC (personal opinion). Bill> I totally agree with server application framework classes first and OO RPC much later. From this section, I would add the following items for OpenVMS to pursue: - DCE RPC. Extend to support txRPC standard. Add support for C++ name mangling. Bill> To the best of my knowledge the above is already done and shipping unless txRPC is not what I think it is. - DECdtm. Integrate with txRPC. Support whatever interoperability is required (this is still a devloping area, so the I14Y needs will probably evolve with it). Bill> I have to admit not knowing much about DECdtm. What would be needed on NT to I14Y with DECdtm? Is RTR a possibility in this space to tie together NT and OpenVMS? >>> Assuming some variant of the presumed-abort 2PC that is conducive to an >>> RPC model, what should be required is a foreign transaction manager >>> module that would act as a gateway to the other commit wire protocol. >>> >>> RTR is not really an answer here, although it does make use of DECdtm's >>> foreign transaction manager interface. RTR is a queued model, and even >>> though most applications that I know of using RTR expect a reply soon >>> after the request is entered, the entire design around RTR is geared >>> towards delivering the request regardless of client, server, or routing >>> failures. >>> >>> Actually, there was an interesting problem area that we discussed during >>> the TP days that might be relevent here. I don't know if this has become >>> a real problem for our customers yet. The difficulty with normal atomic >>> commit protocols is that they are blocking under certain failure >>> conditions (even the so-called "non-blocking" ones). One outcome of this >>> is that there is a very strong trust relationship between the various >>> components in a transaction (due to denial of service opportunities). >>> The needed level of trust is often stronger than what is available, >>> especially if considering inter-company transactions. There was this >>> idea kicking around a few years ago that such transactions should be >>> modeled (and implemented) as queued escrow operations, thus keeping the >>> "other party" out of either side's data. That sort of system could be >>> done using a normal transactional RPC for the local processing, and RTR >>> to link the two parties together... - Class libraries. We should provide basic class libraries for a server structure based on the traditional transaction processing structure of an application layer that operates against a suite of resources via recoverable resource managers. This is the application model inherent in the DECdta architecture, fwiw. The opportunity for class libraries includes both those that encapsulate the common server application structure, as well as those that encapsulate most of the recoverability attributes. See the Avalon/C++ work from CMU some years ago, or look at what the Argus V2 project was investigating. - Transaction recovery log manager. Build a simple to use, moderate performance, recovery log manager that would be used by the recoverable resource managers above to implement transaction recovery. Bill> For the above 2 is there a way to provide them and get both performance on openVMS clusters and not lock the customers into OpenVMS? What would be needed on NT? >>> We would need to either have access to, or build, a transaction manager >>> and a log manager on NT. The transaction manager needs to include the >>> resource manager interfaces. >>> >>> Today this would probably involve building both of them, however I >>> believe that Microsoft is currently working on their own 2PC protocol. >>> Again, if we could get information on that interface we could just work >>> with it, rather than try to bulid something in competition. >>> >>> So, no, there's nothing that needs to lock the class libraries into VMS. >>> The actual support routines probably would be VMS specific, but as we >>> should encourage the use of the class libraries, this needn't compromise >>> the portability message. - Threading. Support the switching of default transaction context on a thread switch. This allows a multi-threaded client to use transactions as a recovery mechanism fairly transparently. - Lock Manager. I seem to remember something that may be fixed. If it is, then this is a no-op. If it isn't, this is something to fix. It used to be that the lock manager assumed that all locks in a process were under a single authority's coordination. So it would do things like immediately deadlock if a process was waiting on a lock that it already held. This model never really worked, even before threads, and certainly doesn't work now. Again, if this has already been fixed, great. Bill> Can someone from the cluster group comment? - Investigate DMQ or ACMS. Use them as examples to learn how to efficiently and simply compose server application routines to create larger server applications. If you can get the failure handling right (and using transactions is a big step on that path), this give you a nice line on improving code and design reuse in the servers. Bill> This whole area seems is ripe for Steve Jenkins group to help us with. Do you agree? >>> Yes, there's certainly a lot of existing expertise in the company that >>> we should be able to call on. 2.3 Security Services For security, I'd most like not to see it but have the security behave as if the complet client/server application executed on one system. For instance, I'd like the client's identity authenticated automatically at service connection, I'd like all operations (unless I specifically say otherwise) to execute under that persona, I want all audit records (again, unless I say otherwise) to be recorded against that persona. I may even want to see that persona show up in things such as SHOW USER. Bill> This seems like something we should be able to hide inside the generic server infrastructure and for the most part server applications shouldn't have to worry about. >>> Yes, I'd expect the security support to be hidden within the server >>> infrastructure. >>> >>> The cases that don't fit that model are the management support changes, >>> such as support in SHOW USER to show clients actively connected to an >>> active server. This is admittedly of secondary importance, but is >>> something I'd imagine we'd want to be able to support in the future. And I don't want this at the cost of loss of scaling as the number of threads that could run increases. Again, a lot of the work is already underway. For completeness, my list is: - Impersonation. The must a set of documented services that allow a server to switch its security context on the fly. These services have existed for a long time, so this is mostly productization. - Threading. The threading subsystem should be able to identify a security context with a thread and manage the impersonation switch at thread switch automatically. - Authentication. There should be a service that provides support for an application to determine the identity of the user on an inbound link, and to produce the security persona for later impersonation calls. - DCE RPC. Should make use of the above services so that incoming client connections can be multi-threaded in a server, with each thread holding the right security context. - Testing. All stallable system services must be verified for correct behavior if the security context (including the accessor's identity) is changed during their execution. - (For completeness) $GETJPI, SHOW USER, etc. These services and utilities should be able to display the identities of the various active threads in a given server. As we move more and more to providing servers for a LAN, we'll see fewer processes and direct connections on our systems. As a result, we'll need something like this to find out what work is actually underway on a system at any given point in time. 3. Resource Control Needs This section describes the needs around resource management. How to predictably gate server activity, and how to scale the server's performance as hardware resources are added. 3.1 Intra-server Flow Control Given that the number of clients can grow arbitrarily large, it's important that a server application not suddenly be pushed into thrashing by a small network configuration change. In the past I've found myself handling this by limiting the number of threads that could be actively processing a request and queuing all others that were waiting. While this works, I've not had the sort of monitoring tools to make an intelligent decision and in the end had to guess at a good number (5 seems to work for a lot of cases I've run across -- I don't know why). There's also an opportunity that I've never exploited, but seems like it could be useful. A server will typically do pretty much the same thing over, and over, and over, and... So, it should be possible to pre-package the resource use for a request and just keep recycling it. Just a wild idea, but... Bill> This is definitely a big performance gain but hard to keep transparent to the application. So, this time I've got very few specific proposals. Those I have are: - Monitoring tools. We need monitoring, tuning, and debugging tools that are as effective at spotting bottlenecks on a per-thread basis as we have for processes. For instance, the features of MONITOR SYSTEM and MONITOR MODES should be available for threads. Bill> I am not sure what the right technology to use is but we do need an efficient way to debug and performance monitor/characterize complex client/server applications. >>> Agreed. - Instrumentation. If you use the base classes above, the monitoring and status reporting should improve by allowing the class implementations to feed extra data to the MONITOR subsystem. For instance, the ability to monitor internal server application queue lengths, transaction log use, and lock conflicts would be very helpful for server application designers and (potentially, depending on server implementation) system managers. - DCE. The DCE subsystem should support any monitoring and instrumentation that would allow a server application to efficiently locate any perf. related holes. My thought is that things like server application queue length would be instrumented in the RPC runtime as opposed to the server application itself. (Btw, I'd consider shipping a developer's runtime with a bunch of checks and monitoring points on, and a stripped down racing version.) 3.2 Scaling This means threading, but to be frank, I'm not sure that I completely buy that kernel threads will be much of a help. That's probably because the servers I've built in the past were I/O bound, when they were bound at all, so allowing a second processor to help the execution wouldn't have mattered. I'll grant you that other servers, such as database servers, will probably run differently. The only other route to scaling is to support deployment of servers in parallel, and to split the work across them. If we could come up with some management services that made that easy, even if it involved further splitting of the application's data partitions, it would help show a unique and powerful feature of OpenVMS -- namely that I can get shared nothing performance at the application, and use the shared everything base to improve scaling and add availability. Soffex, and the other stock markets that we automated use this sort of system -- they deploy a set of servers that are each responsible for a subset of the stocks. If a server fails, or comes online, the subsets are recalculated (well, it's not quite that fancy), and the load rebalanced. If we could provide some set of services that helped an application do that, and could explain when you'd want to use them, we'd have something unique... Bill> I agree that for scaling the application needs to be able to partition the data. I don't have any specific proposals though. 4. Configuration Needs Next, this section describes configuration issues. This includes the question of placing the servers, how (and whether) to load balance, and what could help on server failover and failback. Note this last is an area where OpenVMS clusters could provide a unique opportunity. We can today build servers that provide the performance during normal running that you'd expect from a partioned data model, and the availability you'd expect from a shared data model. 4.1 Server Placement This isn't the issue of looking up a server, but rather the config control needs of where a server is best placed, where to place its standbys, when to failback, how to determine if the loading is incorrect, etc. Today this is a very manual task, and often fraught wiht unforseen consequences. I'd want to see some sort of monitoring and advisory tool that helped the system manager understand where a server's load would be "best" located in a cluster. For instance, a tool that looked at the I/O bandwidths, CPU and memory capacity of the various nodes, their current loads, and the loading from a given server, and suggest placement alternatives would be very useful. Also, I'm sure our customers would find such a tool invaluable for "what if" scenarios involving disk, memory, or CPU upgrades. Could Polycenter do this, and/or does it have plans to do something like it? Bill> Resource Broker has developed this technology, PATHWORKs to some extent as well. It needs to be packaged up and tied off for use by server apps. >>> Good! - Documentation. In the absence of some configuration tool, such as the one described above, OpenVMS must provide some form of documentation offering guidelines for efficient and robust placement of server applications. 4.2 Failover/Failback Services As you can probably tell by now, I firmly believe that there is significant value in OpenVMS taking a strong view towards server application structure. This not only allows us to offer more complete services for applications that follow that structure (e.g. the monitoring tools above), but also allows us to focus on areas where OpenVMS can truly offer excellence. The first such area that springs to mind is in high availability. Now, I'm convinced that we should learn from the tools that our competitors have used in the past in arenas where high availability was of extreme importance. That means we should learn from IBM's work with IMS/XRF, from Tandem's work with fault tolerant processes, and from countless database research and commercial activities of the past 20 years, as well as our own efforts in projects such as SOFFEX. These lead me inexorably towards server processes, with hot standbys, that use transactions as their recovery mechanism. The sad thing is that we've had most of the underlying support for several years, and never made use of it. We are very well placed to bring availability techniques that have been used at the high end for years downmarket. Aside from Tandem, which is pursuing a similar strategy (hopefully to noone's surprise), this should give us a technological edge. Aside from the class library, transaction management, and log management items above, there are at least the following items: - Clusters. A formalization of a doorbell service that arbitrates which standby server is to be the next primary one. This is both to get a simpler interface, and to cut down on the number of locks such mechanisms produce today. Bill> Cluster group? - IPC/DCE RPC. Notification when a standby is to become a primary. Some sort of ability to have a back-channel from a primary to the secondaries. This latter feature could be used to provide a continuous recovery mode, similar to what is used by IMS/XRF. Bill> I need to think about this some more. I can see a benefit to creating a second server and moving connections to the new server with no user level disruption to allow a node to be brought down or a new version of the server to be brought up. >>> I think I've managed to mis-explain this. IMS/XRF works by having the >>> user connect to a terminal controller that is connected to a pair of >>> mainframes. The controller passes the user's requests to the mainframe >>> designated as the primary. The primary exchanges keep alive messages >>> with the terminal controller, which allows the controller to fail the >>> user connections over to the secondary mainframe when the primary fails. >>> >>> Back at the mainframe, the application is running under the IMS framework >>> and is relatively oblivious to this infrastructure. It processes input >>> requests, updates values in the IMS database, and issues replies. The >>> IMS database manager, on the other hand, is quite aware of the >>> infrastructure. The current primary is taking transactions, and >>> mirroring the transaction recovery logs -- one to the actual recovery log >>> and one across the channel-to-channel adaptor to the backup processor. >>> The current backup is running recovery continuously using the recovery >>> log arriving across the channel-to-channel adaptor. This allows the >>> database to recover virtually instantly, and to produce failover times of >>> less than 30 seconds (this is the interruption length until new requests >>> are accepted and processed). >>> >>> Anyway, this use of a channel between primary and secondaries to help a >>> recoverable resource manager speed its recovery and/or failover times is >>> what I meant by a backchannel. This secondary channel shouldn't be >>> visible to the application proper, nor to the client layer. - Class Libraries. Incorporate server arbitration services, and recovery mechanisms into the base class library's implementation. Most of this is very mechanical and could be completely hidden. Bill> These are the types of things that would be very useful to app developers especially if we make them available on NT as well. - Documentation. Describe how to build a highly available service, how to handle failover, failback, recovery. This should also discuss basic design philosophies (one that bites nearly everyone at first is that there should be as close to no special recovery code as possible). Bill> Agreed. - Finally, go and build one. Find out what else is missing. Add that to the list. If I had to pick one, I'd suggest building the AUDIT_SERVER as a new shared-nothing cluster-wide server with hot backup. Others on my list include the other I/O bound security services (authentication, rights list manipulation, low-level uaf manipulation). As a final note I'd add that this is completely orthogonal to the question of migrating processes. We should be careful not to confuse the issues when we discuss them. Bill> Say more. The technology is completely different. This approach requires participation of the app writers but provides additional benefits too. >>> Process migration is basically about moving an application and its >>> operating environment from point A to point B. The assumption, or >>> requirement, is that the operating environment doesn't actually change >>> between premigration and postmigration times. This is a direct >>> implication of a process migration service that is transparent to the >>> application. >>> >>> However, this still leaves open the question of dealing correctly in a >>> distributed environment with failures of one of more components during >>> normal execution. The easiest way to deal with that is to base the >>> application execution on distributed transactions, and build the resource >>> managers (and infrastructure) in such a way as to make recovery from >>> unplanned failures as close to recovery from planned failures (user ABORT) >>> as possible. This should leave a very small increment of design to >>> implement a failover capability, and an equivalent increment for a >>> reasonable planned failback capability. >>> >>> Finally, while it is possible to build a failover service using process >>> migration, this does not release the application program from having to >>> put in place the same recovery mechanisms for dealing with the final case >>> of the failure of the last running service, and its restart. 5. Management and Operational Needs An area we often forget, but is very necessary for a real solution, is that of server management and operations. This is probably a lesser thought out section, and is used to raise the point that some of OpenVMS' current management structure is geared to usage patterns that are simply incorrect for server applications. For instance, we're currently used to direct connections with the process boundary holding much of the accounting, identification, and security information. We're used to seeing big systems with hundreds of process slots in use. Yet, if we're successful with server systems (and in some markets we already are), the systems will contain very few active processes and the "users" will be connections into those processes. The thread boundary, and not the process boundary, will demarcate the above information. Frankly, I've not had the time to think through the implications of this. I've included a couple of work items that I believe are obvious from the changes listed above. This list is clearly not complete. My minimal list is: - SNMP extension API. There should be at least one documented API for building SNMP extensions. This is based on the presumption that servers in a networked environment should be manageable remotely. Note that we should really attempt to encourage the support of this API from more than just our SNMP agent. Logic dictates that the API we support should be one of the ones currently in the running to become a defacto standard. Bill> I believe we should be using BackOffice APIs to support remote management of applications instead of SNMP to better fit in with the LAN. >>> Fine. Whatever works in the target environment. - Argus. It should become possible for an application to extend the features and capabilities of Argus to incorporate application-specific rules and data. For instance, using ACMS as an example, it should be possible for UA management to transparently include the data fields that are needed by ACMS. Bill> Agreed. - Accounting. Per-thread, per-impersonation context, resource accounting needs to be provided. By going from a timesharing to a server system, we have essentially yanked support for any sort of resource accounting or tracking. Regardless of whether or not we think users will perform chargeback accounting, the inability to even track usage affects the system manager's ability to tune, react to break-in attempts, or any number of other activities. Bill> Lower priority in my mind but it will become important if we are ever to charge or allow customers to charge for client usage. >>> Actually, I wasn't looking specifically at charging for usage. This was >>> just an example of how our tracking data is process based, and following >>> the antics of a rogue client is going to be difficult, at best. Having >>> said that, I'll agree that this is lower priority than just getting some >>> reasonable level of support for building and deploying server applications. [Personal side-note. I've NEVER understood where one data store ends and another begins in the whole resource accounting, performance monitoring, and security tracking arena in OpenVMS. While it never seems to be too high on anyone's worklist, some rationalization there, ideally to a single normalized data store, could only be an improvement for us and for the system manager.] 6. Development Needs The final area to discuss is around the development tools that would be helpful for the development and deployment of server applications. I'm not going to claim any great insight in this area -- it's not an area that I've worked in, and everything in this section should be taken with a grain of salt. What I've tried to do is to list the problems I've faced in the past, and some linear approaches to dealing with them. For all I know these may all be considered "old technology" by now, and already be superseded in someone's plans. The problem areas I've encountered in the past centered on debugging across the client and server, monitoring overall application state, reproducible testing, and underlying programming infrastructure. - Debugging Tools. It's pretty straightforward today to debug a single threaded, single process application. It's more complicated to debug a multi-threaded, single process application -- although supported and possible if you use the DECthreads services (an example of taking a strong view on application structure to improve overall support). However, if you have a distributed application, composed of several processes on various nodes, it is arduous, at least, to perform basic debugging activities, such as taking a breakpoint. It can be done, but it is neither fun or easy -- and it is certainly not natural. Bill> You can use the multi-process debugger today and debug a complete multi-process client/server distributed application on OpenVMS but there is some roll your own involved. Tom Hoey did this to debug a DCE client/server app and all the DCE daemons in 1 debug session. >>> Good. I was hoping things had moved on in the last few years. - Monitoring Tools. Likewise, it's possible through the use of tracepoints and such to monitor the execution of a single process application in a straightforward manner. In fact, the problem I've often had is handling the amount of data I've been able to get, not in getting it. However, there are few tools I've run across (and this is one area that I certainly expect DCE has improved things) that will let me easily build up a global understanding of the execution flow and state of a distributed application. - Testing Tools. Similarly, it is fairly straightforward to build a test harness that will present an application under test a defined sequence of inputs. In fact, if I've followed the description correctly, we now have a tool that automates much of this (JIG?). Tools that monitor message traffic, record it, and then play it back as a regression test framework would be very helpful. For that matter, if the application is using DCE RPC, it would be very helpful to be able to construct a proposed message flow directly. Again, I'm hoping that DCE has such a beast already. - Programming Infrastructure. Finally, I come back again to the use of a "foundation" class library. Anything we can do to simplify the building of distributed applications, and to make building them more natural can only be a win. Now, there's nothing in this list that I haven't worked around in the past, and there's nothing here that will keep an application writer from building and deploying a server application on OpenVMS. However, the result is that the system won't keep you from doing this, but it also won't help you do it either. Consequently, we're going to lose out to systems that do make it easy and straightforward. 7. Final thoughts Now that I've gone through this list I'm struck even more by how much like ACMS classic this looks. I know this is really wild, but could we consider poaching some of the ACMS code base, and either integrating it or putting it into a "business critical server builder" package? That isn't that far form what Tandem have done, and I guess I can now see why. Bill> It does look like we should work with the ACMS group in this whole area. *********************Digital Confidential********************** ================================================================================ Note 5.1 Possible OpenVMS Product Concepts 1 of 2 STAR::BMATTHEWS 72 lines 9-AUG-1995 13:56 -< List of Possible Product Concepts Grouped by Theme >- -------------------------------------------------------------------------------- List of Possible Product Concepts Grouped by Theme Theme : Seamless Access to Data Product Concept: OpenVMS Central Storage Server Product Concept: OpenVMS Integrated LAN Services Product Concept: OpenVMS as a Distributed Object Data Server Theme: Windows NT I14Y Product Concept: OpenVMS Server Component/Extension of MS BackOffice Product Concept: OpenVMS Out-of-the-Box Server Product Concept: OpenVMS/Windows NT Integrated TP Product Concept: OpenVMS/Windows NT Connectivity/Management Theme: System Management of OpenVMS from Windows NT Product Concept: Windows NT System Management Station for OpenVMS Product Concept: Enterprise Mgmt of OpenVMS Theme: Application Development Product Concept: 3 tier C/S Application Development Station Product Concept: Windows NT Application Development Station for OpenVMS Product Concept: Common code for WIndows NT and OpenVMS Product Concept: OpenVMS as a Distributed Object Application Server Theme: Business Critcal Server Product Concept: Integrated OpenVMS and Windows NT Clusters Product Concept: OpenVMS Highly Available Application Server Theme: Dollar Futures Product Concept: File system and backup via CD Jukeboxes and Dollar Product Concept: Data vaulting via continuous backup/restore of Dollar Product Concept: License the Dollar technology Product Concept: Continuous backup/recovery Product Concept: Hybrid inplace file system & log-structured file system Product Concept: Heterogeneous file support Product Concept: High performance data servers based on Dollar Product Concept: OpenVMS clients outside the cluster Product Concept: VAX access to Dollar Product Concept: Improved HSM and Dollar by shelving Dollar segments Product Concept: Storage management via the Dollar cleaner ================================================================================ Note 5.2 Possible OpenVMS Product Concepts 2 of 2 MOVIES::ROBLES "Russell Robles, EDO 13" 26 lines 10-AUG-1995 08:25 -------------------------------------------------------------------------------- > o OpenVMS Server component/extension of MS BackOffice > > > ......The Product concept is to add a component to MS BackOffice to > enhance Windows NT ability to connect and interoperate with > OpenVMS. The idea is that by having an OpenVMS product as part > of BackOffice we would get some Microsoft marketing and would > get Microsoft to ship some sw that enhances OpenVMS connection > to Windows NT. We would also get a large percentage of the > revenue from the sale of the OpenVMS component. > >Product Concepts for the Theme - Application Development > o 3 tier C/S Application Development Station > o Windows NT Application Development Station for OpenVMS > o Common code for WIndows NT and OpenVMS > o OpenVMS as a Distributed Object Application Server Strikes me that you could put these two ideas together. We should be able to put together all the bits needed for cross development of VMS server applications on NT, and arrange with Microsoft to package them with their NT s/w development tools. The goal would be to make sure all/most ISVs can easily port to VMS, and it would in itself be good PR for VMS. We might make some money on the software, but I'd think MS would drive a hard bargain on leveraging their distribution in this way. R.