A Few More Thoughts about Infra

I know, I know, it has been months since I posted.  Personal note: My life got a bit complicated in late February and early March, and then at the end of April, I started a new job.  My work on Infra moved from being my main professional project to being a labor of love that I fit in whenever I can spare some energy from the projects that pay the bills.  So, I'm going to write a few more thoughts on Infra here, and then I'll write a few posts on some items that came out of my current paid gig.

Application Implementation Considerations and Options

With the database design done, engineered to a relational design and some physical implementation scripts generated, my attention turned to the front end implementation of INFRA.

Requirements

INFRA is intended to be available as an application that anyone can install on just about any relational database and just about any application infrastructure. And I don’t want cost to be a major factor for anyone who wants to use it, so I want to look at free to use frameworks and tools. I say “free to use” because I have expertise in Oracle tools and frameworks, and while these are not open source, there are several options that are free to use. Still, many potential users of INFRA will prefer not to be locked into the tools of a single vendor (and I’m afraid they are strongly biased AGAINST Oracle), and thus will prefer open source.
As for requirements, I’d prefer to stick with standards and especially do as much with plain HTML and CSS as possible. I also want good tooling, and am very fond of having the tools do as much of the work as possible. So I want a good IDE with good support for the chosen framework. I have nothing against a good text editor to write short programs where I am hand writing most of the code. For an entire project, however, nothing beats a good IDE, and I have experience with three, NetBeans, Eclipse and Oracle JDeveloper. So here is a little about the choices I considered:

Oracle Application Development Framework (ADF)

ADF has been my professional choice since 2010, and I am still the Lead Moderator for the ADF Enterprise Methodology Group (ADF EMG). So despite the bias against anything from Oracle, I needed to consider ADF, because I can hit the ground running with it. ADF is certainly not open source, which is a mark against it, but there is a subset of ADF called ADF Essentials that is free to use. Unfortunately, the full ADF product only runs on Oracle WebLogic Server (WLS), but ADF Essentials is supported on Glassfish, and is known to work on several other JEE standards compliant application servers, like JBoss and with careful configuration on Apache Tomcat. ADF is a Model/View/Controller (MVC) based framework. It supports the JPA standard for Object Relational Mapping (ORM) for its Model and Java Server Faces (JSF) for its View. The ADF Faces JSF component library has lots of widgets that give you lots of options for layout, data presentation, and input/update forms.
My favorite part of ADF is tooling. You can develop ADF Essentials applications in Oracle JDeveloper, but you do have to be careful to stay away from parts of ADF that require the full tool – i.e. must be deployed on WLS. But JDeveloper has wonderful WYSIWYG support for ADF Faces (and Apache Trinidad) components and can do things like let you drag fields from your ORM onto a page and generate a table component configured to display them, or a form to insert, update or delete data in the database. There have been complaints that JDeveloper is slow and buggy however, and it isn’t in very wide use outside the Oracle user community. As an alternative, you can develop ADF Essentials applications with Eclipse by installing the Oracle Enterprise Pack for Eclipse (OEPE). OEPE doesn’t show ADF Faces components quite as well as JDeveloper, and doesn’t have as much code generation, but it is not bad, and probably a little more stable.
The problems with ADF are: It isn’t open source. Oracle’s support for ADF has been waning, and that means that updates of ADF and of JDeveloper have been very widely spaced. Later versions of JDeveloper have bugs that have not been adequately addressed. It has never gained a large following in the developer community. But to me, the killer that sent me looking for an alternative is that ADF Essentials does not support ADF Security for authorization and authentication of your application. In earlier versions of ADF, this was okay because you COULD configure standard JEE security instead, or use a third party security framework like Spring Security or Apache Shiro. But the latest versions of ADF Essentials broke this, and while I’ve developed a work around, I’m not happy with it.

Other JSF Libraries

Java Server Faces is a JEE standard for the View and Controller, and so is JPA for the Model. Through ADF, I was already familiar with how these work, so a different set of JSF components seemed like a good option. There are a number of libraries around, many of which are open source.

Apache Trinidad

One faces library that appealed to me is Apache Trinidad, which originated as Oracle’s earlier 10g version of ADF Faces (current version is 12c 2.3), so I had actually used this in my first few ADF projects. Oracle opened the source to this library and contributed it to the Apache MyFaces project. This is under active development. One nice thing about Trinidad is that both JDeveloper and Eclipse with OEPE have tooling support for it, so you can get some of the drag and drop and code generation features of these tools without tying yourself to Oracle’s proprietary ADF Faces. Or you can add JSF and the Trinidad library to an Eclipse project without using OEPE.
Unfortunately, Trinidad is also not terribly popular among non-Oracle developers. It has been enhanced to keep up with the latest JSF standard, but not much for functionality.

PrimeFaces 

This is probably the most popular JSF library, with good documentation, good examples and sample code.  There is reasonable support for it in both NetBeans and in Eclipse (I have only used these and Oracle JDeveloper as my IDE.) but there isn't much WYSIWYG support.  Very cool is the ability, like ADF, to have themes, and even to let users switch themes.  And the themes are fairly standard - they are based on JQueryUI and tools for developing them are easily available.  You can even get a good number of themes from PrimeFaces' main developer, PrimeTek.  One interesting thing, is that there are projects for PrimeFaces-like UI that works with Angular - PrimeNG, and with React - PrimeReact.

Conclusion

So this is probably my last post on Infra for the time being. It remains an active project, but I still haven't chosen a UI from one of these or perhaps one of the JavaScript frameworks.  And development has slowed to a crawl.  Hard to come home from a hard day of development, and start up the computer for a private project.  But I've accumulated some nice tidbits on SQL Developer Data Modeler to share with my readers - coming to this blog real soon.

INFRA Database Design - Application Manager View

Finally, we get to the person who is supervising the overall application development effort. While this person has an interest in the other views, she/he has a particular interest in the work of the developers and the various versions or releases of the application system and its underlying application database(s).

Application System

This represents the product of the developers’ hard work, the actual application modules that show the screens that the users want and need to see. It includes both front-end modules that the users see, and the back-end modules that process the user input and prepare the user output. It can include the actual code behind those services that are being custom coded. It also includes code to talk to the various services whether custom written or not, and to the Application Databases.

Application Version

This may also be called an Application Release. It represents a frozen in time set of code for the modules in the Application System, often numbered according to local standards. For instance, release 1.0 might be the initial release of a system, while 1.2 is the second minor enhancement release. Application Versions are also often identified with a “tag” or “label” in the source control system. Task and Bug management software like JIRA often also have a place to record versions or releases. One goal of INFRA is to be able to coordinate its database with those of various Task Management and/or source control systems.

Application Database

This represents the various database structures and objects that contain and organize data used by one or more Application Systems. It may also include database procedural code like triggers. While an Application System can use objects in more than one Application Database and an Application Database can be used by more than one Application System, as a general rule, the relationship is one database to one application, especially for which application may update the database. Nonetheless, I have designed INFRA to be able to handle the less common many to many situation.

Database Version

This may also be called a Database Release. It represents a frozen in time set of objects in the Application Database, often numbered according to local standards. For instance, release 1.0 might be the initial release of a database, while 1.2 is the second minor enhancement release. It is usual to coordinate the database version numbers with the application version numbers – so that version 2.5 of the application goes with version 2.5 of its database. Even if you don’t do this, it is very important to make sure which version of the application matches which version of the database. Database Versions are also often identified with a “tag” or “label” in the source control system.

Entity Relationship Diagram for Application Manager View

This concludes my series on the database design for INFRA.  I am very interested in what you think of INFRA so far.  What did I get right?  What did I get wrong?  Where would you go from here?  Next time, I'll talk about implementation options.

INFRA Users

So who is the audience for INFRA?  I’ve identified four groups of people, each with its own perspective on the overall infrastructure:

System Administrators

These are the people who manage your physical and virtual servers.  They want to see what hardware they have, what operating systems (OS) are on them, what virtual machines (VM) and the services on each server.  A service can be a database, an application server, a web server, an LDAP directory, or maybe a microservice – anything needed to operate your applications.

Database Administrators (DBA)

Databases are essential parts of any application system – I have never seen an application that didn’t have a database behind it.  Early in my career, a database may have simply been a set of files.  Later, we were using Relational Database Management Systems (RDBMS) like Oracle, SQL Server, DB2, PostgreSQL or My SQL – and in fact, that is what I’m designing INFRA to use.  Today, you may see other forms of databases, like Hadoop or Cassandra. The DBAs are responsible for managing these important resources.  And of course, this is my particular area of expertise.

Deployment Managers

Eventually, your application needs to be deployed to the various services that operate it.  This is more complex than it used to be.  Early in my career, you compiled the application and just ran the compiled code.  But now you may have different pieces – REST services to provide access to databases, data to enter into LDAP directories, HTML, CSS, Javascript and image files to write to directories accessed by a web server, or JAR, EAR, or WAR files to install on an application server. You may not have a job specifically designated as “Deployment Manager”, but believe me, you have people doing this job.

Application Managers

These are the people responsible for the overall development effort for your applications.  They determine who will do the development work, and what will be developed.  In particular, these people decide which changes will go into each release of an application, and which release will be deployed in which environment.

About the Database Design

Considering that my specialty has been as a database architect, administrator and developer, it is only natural that I would begin the application design with a database design.  I consider the Logical Design to be an important part of the requirements definition of an application.  With this, I find out what “things” - formally called “entities” we are going to collect information about, what information is going to be collected, and how the entities are related to one another.  And once this is engineered to a Physical Design and a development database is created, many tools can use the database design to give you a head start toward developing the rest of the application.  For instance, you can often use a relational table to generate a first cut at the forms that the application will use to enter or update data in the table, or the first cut at pages that show this data.

And since I have been a DBA, it is only natural for me to start by designing the part of the database that will contain the information that I would most want to see in my view of INFRA.  That is the subject of my next post.

Introducing INFRA – an Application to View Your Infrastructure

On my last two projects, I was the database architect and administrator.  Although our projects were small, we needed to maintain multiple sets of servers, each with different versions of the applications, databases, identity services, and other support services.  We had a development environment, a test environment, a user acceptance test environment, a pre-production (aka staging) environment and of course, production.  While some of these shared servers, others needed separate servers.  Most servers were virtual machines, but not all. So it became difficult to answer questions like:

• Which server is running our development Oracle database?
• Is the test application running against the test database?
• How can we quickly reset the user acceptance test environment for a new round of UAT?
• The backup files for the production SQL Server database need to be written to separate physical hardware. Which servers with room for these files meet this requirement?
• Can we take this server down for an upgrade today?
• What version of the application is on the staging application server?

I’ve been thinking about this problem on and off for several years.  I’ve done some research, and I can’t seem to find a COTS application that stores this information.  There are source control systems that can distinguish between the source code for different versions of an application, and I use this for different versions of database objects.  JIRA, which is an application that we use a lot for task and bug tracking, can list Releases of an application, lets you identify changes for each release, and can read the data in your source control system to tie releases to the source code that was changed.  Virtual Machine (VM) control software used by our system administrators, like the control software for VMWare can identify which virtual servers are running and which physical servers are hosting them.  But nothing ties all of this information together.

So, being between jobs, I decided to start writing a new application to keep the lists of servers, databases, application releases, and other services.  And especially, to keep the relationships between them – what release of the application is on this server, which databases it is using, where are the user ids and passwords that can access this application on this server. The goal is to be able to answer the questions listed above and others like them.

I call my new application, INFRA.  It is a work in progress, but I thought I’d share the process with my readers.  The initial database design is pretty much done, so I’ll start with that.  I’ll be inviting you to share your thoughts too.  Tell me where I’ve gotten it right, where I’ve gotten it wrong.  Tell me if the whole idea is stupid – I’d love to hear that someone has already solved this problem.