KScope '17

KScope '17
I'm speaking at KScope '17.

Friday, June 16, 2017

Update on Reading your ERD

This is just a quick update to my earlier post Reading Entity Relationship Diagrams to Users.  In it, I talked about how to read ERDs so users can understand them and how to use this to draw out more information about their entities and particularly, relationships.

These ideas are not original with me.  I first learned them as part of my education in Oracle's CASE*Method, and implemented in Oracle's old product, Oracle Designer.  The data modeling tools, like Oracle SQL Developer Data Modeler, that can show Barker notation for ERDs are also drawing from this way of thinking about data modeling.  Richard Barker was the creator of CASE*Method and lead the team that created Oracle Designer.

But I am especially indebted to Mike Lynott, formerly a consultant to Oracle, and currently Customer Solutions Architect at eprentise (according to LinkedIn).  It was his white paper, Why Do We Say "Each"? that inspired my thinking on the subject.  I finally located my copy of this paper.  I wish I could give you a link to go read it yourself.  It is a little old, but still very relevant to what RDBMS designers do every day.

Thursday, May 18, 2017

Javascript Frameworks

Continuing on with a few opinions about UI design, development and implementation ...

So what is "cool" now?  Javascript frameworks like Angular, Ember, React and Oracle's own JET. These are great, and they have their place, I'm sure.  The argument is that since these frameworks use the power of the user's browser, they can be more responsive.  There is less communication between the server and the client, which should also speed things up. They can do exactly what the designer wants them to do - no component framework getting in the way, wanting to do things its way.  They are open source for the most part, which frees us from Oracle's tyranny.  And working in HTML and CSS can let you use graphic designers and their toolboxes, like DreamWeaver.

The first problem I have with Javascript frameworks is that this is a return to code it yourself.  The tooling builds some of the code as sample files, where you take some prebuilt code and tweak it.  For the most part, however, you have to write the HTML, the CSS, the Javascript.  Fortunately, there is some help in the libraries, like JQueryUI, with some precoded components, and lots of sample code.  But I miss JDeveloper's ability to drag a data control onto a page, say you want to drop it as a bar graph, do a little configuration, and presto.

Tooling and IDEs

Of course, in some respects, this is a problem with tooling rather than with the framework.  I suppose that if I could point Eclipse or Netbeans at a REST service and have it generate the code for say, an Angular 2 service module that accesses the service, I might feel differently.  Or if I could generate a first-cut form to do CRUD with that service, I'd feel better about these frameworks.  I also get the feeling that developers WANT to code it, and hate handing over part of the job to someone else, especially with something like ADF, where you can't easily see the source code.

The proponents say, "but you can make it look EXACTLY the way you want."  True, but at what cost? Can your users wait while you do it?  But you say "It's Agile, we'll do 80% now and get it closer to 100% with each iteration."  And I reply, "I can give you 98% in the time it takes you to do 80%, and do the users care if I can't get to 100%?"  If they DO care, guess what, ADF 12.2.1 uses JSF 2.2 (and so do the latest versions of other Faces libraries), and you CAN code in HTML5/CSS/Javascript what you can't do with JSF, yet still interact with the pure JSF parts. Someday, I'd love to do a contest - who can meet the user's requirements fastest.

Responsiveness and Speed

Another advantage claimed for these environments is that most of the processing takes place on the user's computer, and interacts directly with the browser.  Since the application doesn't have to interact with the server as much, it runs faster.  And on the server side, it isn't as complicated and doesn't require as much server resources.  All the server is doing is downloading files to the client, and running REST services.  What's more, the browser usually caches files that it has already downloaded, so if the file hasn't changed, it will use its local copy of the file, and won't go to the server for a fresh copy.  Please note that this can be a disadvantage for mobile devices with slow connections and limited storage.  There is a lot more code to be downloaded and stored.

The honest truth is that in ADF Faces many of the Rich Faces components are rendered using Oracle JET, so they have all the responsiveness of this Javascript library, except that I didn't have to code it.  Similarly, PrimeFaces renders with JQuery and JQueryUI.  Because PrimeFaces does this, it can use JQuery tools to create themes.  ADF Faces uses skinning through CSS-like files and the basis for its skins is in the open source Apache Trinidad Faces library.  Of course, this means that JSF frameworks can be subject to some of the same problems as the Javascript frameworks that they use for rendering.

Testing and Security

One problem that I have with Javascript frameworks is that all of the source code is delivered to the user's browser.  Javascript is interpreted code, which means that some bugs won't show themselves until runtime.  Compilers can catch some problems up front, which can speed development.  I should note that Angular 2 & 4 are usually written in Typescript which compiles to Javascript - a little of the best of both worlds.  But the biggest problem I have with this is the advantage it gives to hackers.  When a hacker can see ALL of the source code, including code that is intended to keep bad data out of the database, he/she can re-write it to bypass your safeguards.  If you are going to write REST services for your Javascript framework to use, you need to make sure that you have server-side protection.  Never accept data from a browser without validating it at the server.  Don't assume that the Javascript code that you wrote will be executed unchanged.

So there I am – an old developer who found a great way to develop and doesn't want to give it up.  I'm not against the Javascript frameworks, and if you want to build applications that way, good for you.  I'll do my best to support you.  I know enough HTML and CSS to be useful, and even some Javascript.  As a long-time database advocate, SQL and PL/SQL expert and developer, I'll write good solid secure REST services for you.  Does that make me obsolete?  I don't believe so.  I'm not quite ready to leave you without my help and advice yet.

Monday, May 15, 2017

User Interfaces - a Personal History

I've been doing user interfaces (UI) for a long time.  I hate to say how long, because I'm looking for a new job, and I'm afraid that age discrimination is very real.  And I'm also aware that behind my back some of my younger colleagues think that I'm set in my ways, behind the times, and in short, obsolete.

A little history

To be sure, my first user interfaces were a sort of template for users to enter data into a fixed record format file,  The only validations were to designate fields as alphanumeric or numeric.  My next applications let me draw forms on a terminal, and that gave me the ability to write (in COBOL of all things) code to validate and then write data to data files.  I graduated from there to Oracle Forms.  What a difference!  Now I could point my tool at a relational table, even several tables in master/detail relationships, and it would generate a first cut at forms to update those tables.
Oddly, my next step was a step back - I was working in HTML and once again had to develop my own forms and write my own code to process the input and decide which tables to update and what to feed back to my users.  At least PL/SQL is an easier language for that than COBOL.  In about 2000, it was nice to get a hold of Oracle Designer, and its web generation capability, which would create forms for me.  But then, I lost a lot of control over what the form looked like - unless I wanted to take COMPLETE control and not be able to regenerate the form from Designer,

Application Development Framework

Which brings me to Oracle Application Development Framework (ADF),  ADF Faces, the view and controller part of ADF is based on Java Server Faces (JSF), which is part of the Java Enterprise Edition (JEE) standard.  Oracle JDeveloper, the primary IDE for ADF, can generate first cuts at forms, but not just forms: ADF can do tables, graphs, hierarchies, trees, calendars and other data visualization.  It can drill down.  It can navigate from page to page, and this flow of control is shown graphically so that even showing it to a user, the user can get an idea how it is going to work.
A lot of the components have built-in functionality that I don't have to code.  For instance, an ADF table component can let users reorder columns, re-sort, and filter, with no code.  There are tabbed layouts, split screens (vertically or horizontally), and accordion layouts.  There are popups and quite a bit of control over when a popup is displayed and when hidden.  Some layout components are especially designed to rearrange themselves according to the size of the screen - showing differently on a phone than on a desktop for instance. The newest version of ADF can inform the application about the screen dimensions so that the application can decide what components to show.
ADF can do single page applications where the header, main menu and footer are all static with only the content in the middle changes.  It can do AJAX, changing only the parts of the page that need refreshing.

Other JEE/JSF Frameworks

One of the arguments against Oracle ADF is that it is Oracle.  And it is not open source – if you are on a support contract, you can get the source code, but you will need to sign a non-disclosure agreement and you can't change it.  Many people would prefer to use open source software only, and avoid depending on Oracle.

I can't argue with the advantages of going the open source route.  ADF Essentials lets me use ADF for free, and I can use an open source JEE application server.  And I've written an ADF application against an open source database, so we don't HAVE to use Oracle's expensive RDBMS.  But even ADF Essentials is NOT open source, and it doesn't do everything the non-free version can do on the non-free Oracle WebLogic Server.  We can get away from ADF BC and use an open source JPA implementation - and if you write ADF Essentials applications with Eclipse, you probably will do that. To get away from the ADF Controller and ADF Faces to an open source replacement, you can use the base JSF controller and an open source JSF library like PrimeFaces.  But in doing this, you will lose some of the wonderful tooling that JDeveloper has for code generation.  Or even Oracle Enterprise Pack for Eclipse has pretty good tools for developing ADF Essentials applications with EclipseLink for the Model and ADF Faces for the View and Controller.

I have also used some of the tools in Eclipse for developing PrimeFaces, and it is better than using the Javascript frameworks, but not as good as ADF Faces in JDeveloper.  In my next post, I'm going to talk about the Javascript frameworks.

Tuesday, May 2, 2017

Generating REST from the Oracle HR Schema

More and more web applications are being written with HTML, CSS, and (especially) Javascript.  There are a good many frameworks out there – like Angular2 (4?) and EmberJS – for doing this.  The goal of these frameworks is to have the client, the user's web browsers, do most of the work, rather than the server as in my JEE/JSF-based Oracle Application Development Framework (ADF) applications.  They also want to give the user a more responsive, and eye pleasing interface.  But ultimately, they do have to talk to the database, and update the data that is shared with other users.  For this, they tend to use web services, particularly Representational State Transfer (REST) services, which take advantage of the capabilities of the Hypertext Transfer Protocol (HTTP) beyond what is usually used to serve pages over the World Wide Web.

ADF Business Components as REST

I've been playing with various methods to expose my database as REST services, and I've been using the HR schema that comes as a sample schema with every Oracle database as my test platform.  Since I'm primarily an ADF developer right now, the first thing I tried was ADF 12.2.2's greatly improved ability to expose ADF Business Components (ADF BC) developed from the database as REST services.  This works quite well, and is very nice for its ability to use your customizations and View Criteria.  But I'm especially interested in using ADF Essentials and deploying to open source application servers like Glassfish.  And as far as I can determine, REST Web Services from ADF BC is not part of ADF Essentials, so you can only deploy these to Oracle WebLogic Server.

Oracle REST Data Services

Another alternative is to use Oracle REST Data Services (ORDS), which can be deployed to most JEE application servers, including Glassfish and Apache Tomcat.  ORDS even contains an embedded JETTY application server, so it can be deployed standalone for light use.  While ORDS is not open source, it is free to use as part of your Oracle RDBMS license.  ORDS works fine and you can use SQL Developer to expose your tables through REST on ORDS.  I'm not totally comfortable with exposing tables directly, so I would suggest using a three-schema architecture and exposing updatable views based on your tables, rather than the tables themselves.  I did have a little trouble with ORDS configuration, however, and I need to work on this some more.  I will probably write another post about my experiences with ORDS.

JAX-RS – Java REST Services from JPA

Java web developers tend to use Java Persistence Architecture (JPA) and one of its implementations like Hibernate or Eclipselink to do Object-Relational Mapping (ORM) from their relational database to Java objects.  Even ADF developers who write their applications in Eclipse instead of JDeveloper with ADF Essentials will usually use JPA instead of ADF BC.  And there is a standard API called JAX-RS for exposing JPA entity classes as REST services.  The nice thing is that both the Eclipse IDE and the NetBeans IDE have plug-ins that can generate JPA entity classes from tables in the database, and can even generate the code for exposing these as REST services.  I've tried both and while I prefer NetBeans, Eclipse works just fine.  Here is a link to a NetBeans tutorial I used: https://netbeans.org/kb/docs/websvc/rest.html.

I generated RESTful services from the DEPARTMENTS, EMPLOYEES and JOBS tables in the HR schema and deployed to Glassfish.  The JOBS service worked fine, and the count services giving the number of rows in the tables worked for all three tables.  But the other services didn't work for DEPARTMENTS and EMPLOYEES.  It took me a while to figure it out, because it gave a 500 error without putting error messages in the Glassfish logs.  I finally found some information on StackExchange that told me that I needed a class implementing ExceptionMapper<Exception>.  Here is my class:

 * To change this license header, choose License Headers in Project Properties.
 * To change this template file, choose Tools | Templates
 * and open the template in the editor.
package com.smdi.johnf.hr.service;

import javax.ws.rs.core.Response;
import javax.ws.rs.ext.Provider;

 * @author JohnF
public class ExceptionMapper implements javax.ws.rs.ext.ExceptionMapper<Exception> {
    public Response toResponse(Exception exception) {
      return Response.status(500).build();

Ahhh – with this class to tell it to print a stack trace upon an error, I could finally see what was wrong.  The problem is that there are circular references in DEPARTMENTS and EMPLOYEES.  DEPARTMENTS, for instance has a column named MANAGER_ID, which references the EMPLOYEE_ID of an EMPLOYEES row for the department's manager.  This in turn has a DEPARTMENT_ID for the DEPARTMENT for which the employee works and its own MANAGER_ID for the employee's manager.  When generating JSON or XML for the service, the API doesn't have a way to stop.  One answer for DEPARTMENTS is to add the @XmlTransient annotation to the List of Employees in the department – which tells the API not to continue to dive deeper into the hierarchy of related entities.

Hope this exploration of creating REST services for your database is useful.

Friday, April 21, 2017

ODTUG KScope 17

Just a quick note to let my readers know:  I'm speaking at ODTUG KScope for the 16th time - I've done a presentation at every ODTUG conference since 2002.  This year, I'm presenting "Updatable Views to Secure your Database and Make your Developers Happy" which encapsulates and updates a few posts I did on my old blog - http://it.toolbox.com/blogs/jjflash-oracle-journal, last fall.  Come see me at the J.W. Marriott San Antonio Hill Country on June 26, 2017.  I'll be presenting in the Azalea room at 10:30 AM.  Do step up and introduce yourself if you see me at KScope - I'd love to meet you.

Unfortunately, I will only be at the conference on Sunday and Monday.  For various reasons, my company was not able to support my attendance this year, so I'm footing the bill myself.

Thursday, April 13, 2017

Triggers, Jobs, Events, and Queues - Part 2

In my previous post, I talked about the problem of triggers that do things outside of the current transaction.  My example was using the HR sample schema.  The problem: send the department manager an e-mail whenever a new employee is added to the manager's department, or an existing employee is transferred.  The first impulse is to write a trigger on the EMPLOYEES table, but this doesn't do what we want.  Either we get a mutating table error because we are trying to SELECT from the table that we are in the middle of updating.  Or if we make it run a procedure in an AUTONOMOUS TRANSACTION, the e-mail gets sent even if we roll back the changes.

So my first solution was to use DBMS_JOB to start a job to send the e-mail.  Since the job runs in a separate session, it doesn't get the mutating table error.  And since the job is not submitted until the current transaction is committed, the manager does not get an e-mail unless the changes are saved. But there is another way.

Method 2:  Advanced Queuing and DBMS_SCHEDULER

While the DBMS_JOB method is simpler, and I honestly would use that method for the given use case, there are two problems with it.  First of all, DBMS_SCHEDULER is preferred over DBMS_JOB for scheduling jobs, and I would not be surprised if Oracle were to deprecate DBMS_JOB someday.  But DBMS_SCHEDULER runs immediately, and does not depend on a COMMIT, so we can't use it directly.  Secondly, DBMS_JOB would be inconvenient for more complex  requirements, for instance to conditionally trigger a whole chain of actions on a table update.  DBMS_SCHEDULER is great for more complicated requirements like this.  How can we use it to schedule work to be done upon saving changes to a table, since it doesn't require a COMMIT?

The answer is to combine DBMS_SCHEDULER with Oracle Streams Advanced Queuing (AQ).  With AQ, your table trigger can raise an event by enqueuing a message, and the message is not enqueued without a COMMIT.  Besides being able to schedule jobs based on time, DBMS_SCHEDULER can also schedule jobs that run when an event is raised.
So first, let's create a queue to hold the information that needs to be passed to the employee_dept_chg_email procedure.  To do this, we need to grant privileges to some users.  I like to have a user whose purpose is to create and administer queues, then have this user grant privileges on the queues to other users, like our table owner, HR.

GRANT aq_administrator_role TO aq_admin;
GRANT aq_user_role TO hr;

Then as AQ_ADMIN, I create a TYPE to hold the information that needs to be passed to the procedure, a queue table with the type as its payload, and a queue based on the table:

CREATE OR REPLACE TYPE employee_dept_chg_type AS OBJECT (
  department_id NUMBER(4),
  first_name    VARCHAR2(20),
  last_name     VARCHAR2(25),
  job_id        VARCHAR2(10)
/* Multiple consumers must be true to use this queue to start jobs. */
EXECUTE DBMS_AQADM.create_queue_table ( -
   queue_table            =>  'aq_admin.employee_dept_chg_tab', -
   queue_payload_type     =>  'aq_admin.employee_dept_chg_type',
   multiple_consumers     =>  TRUE

EXECUTE DBMS_AQADM.create_queue ( -
   queue_name            =>  'aq_admin.employee_dept_chg_queue', -
   queue_table           =>  'aq_admin.employee_dept_chg_tab');

EXECUTE DBMS_AQADM.start_queue ( -
   queue_name         => 'aq_admin.employee_dept_chg_queue', -
   enqueue            => TRUE);

Finally, I grant privileges on the type and the queue to HR plus roles and privileges needed for DBMS_AQ and DBMS_SCHEDULER:
GRANT EXECUTE ON employee_dept_chg_type TO hr;
EXECUTE DBMS_AQADM.grant_queue_privilege ( -
   privilege     =>     'ALL', -
   queue_name    =>     'aq_admin.employee_dept_chg_queue', -
   grantee       =>     'hr', -
   grant_option  =>      FALSE);

Now HR can rewrite the trigger to enqueue a message:

CREATE OR REPLACE TRIGGER employees_manager_change_email
  ON employees
  employee_dept_chg_msg aq_admin.employee_dept_chg_type;
  l_enqueue_options     DBMS_AQ.enqueue_options_t;
  l_message_properties  DBMS_AQ.message_properties_t;
  l_message_handle      RAW(16);
  IF :NEW.department_id IS NOT NULL AND
      (UPDATING AND :NEW.department_id <> NVL(:OLD.department_id,0))) THEN
    employee_dept_chg_msg := new aq_admin.employee_dept_chg_type(
    DBMS_AQ.ENQUEUE (queue_name         => 'aq_admin.employee_dept_chg_queue',
                     payload            => employee_dept_chg_msg,
                     enqueue_options    => l_enqueue_options,
                     message_properties => l_message_properties,
                     msgid              => l_message_handle

Enqueuing a message requires a COMMIT, so if the transaction in which the trigger is fired is rolled back, the message is not enqueued.  This is what we want – no e-mail sent unless the change to the employee's manager is committed.
We need to change the procedure a bit to use the message type as a parameter instead of the individual parameters.

CREATE OR REPLACE PROCEDURE employee_dept_chg_email (
  msg  IN aq_admin.employee_dept_chg_type
  manager_first_name  employees.first_name%TYPE;
  manager_last_name   employees.last_name%TYPE;
  manager_email       employees.email%TYPE;
  SELECT   first_name, last_name, email
    INTO   manager_first_name, manager_last_name, manager_email
    FROM   employees
   WHERE   employee_id = (SELECT   manager_id
                            FROM   departments
                           WHERE   department_id = msg.department_id);

  send_mail (
    p_from      => 'HR',
    p_to        => manager_email,
    p_subject   => 'New Employee in your Department',
    p_text      =>    'Please welcome '
                   || msg.first_name
                   || ' '
                   || msg.last_name
                   || ' to Department '
                   || TO_CHAR (msg.department_id)
                   || ' as a '
                   || msg.job_id
                   || '.'
END employee_dept_chg_email;

Now we create the DBMS_SCHEDULER job that will be run whenever a message is placed in the queue.  This is done in several parts.  We create a schedule, then a program to reference the procedure that sends the e-mail, and then the job itself referencing the schedule and program.  The reason to create these separately is the program is going to get its parameter to pass to the procedure from the payload in the queued message.  And we also want to set an attribute on the job to tell Oracle that each message should start a new instance of the job, because the default is that if a job is already running, SCHEDULER shouldn’t start the job again.

  DBMS_SCHEDULER.create_event_schedule (
    schedule_name   => 'hr.employee_dept_chg_schedule',
    start_date      => SYSTIMESTAMP,
    queue_spec      => 'aq_admin.employee_dept_chg_queue'
  DBMS_SCHEDULER.create_program (
    program_name          => 'hr.employee_dept_chg_pgm',
    program_type          => 'STORED_PROCEDURE',
    program_action        => 'hr.employee_dept_chg_email',
    number_of_arguments   => 1,
    enabled               => FALSE,
    comments              => 'Sends an e-mail to notify department manager of new employee.'
  /* This says to get the first argument for the procedure from the event
   * message. */
  DBMS_SCHEDULER.define_metadata_argument (
    program_name         => 'hr.employee_dept_chg_pgm',
    argument_position    => 1,
    metadata_attribute   => 'EVENT_MESSAGE'

  DBMS_SCHEDULER.enable (name => 'hr.employee_dept_chg_pgm');
  DBMS_SCHEDULER.create_job (
    job_name        => 'hr.employee_dept_chg_job',
    program_name    => 'hr.employee_dept_chg_pgm',
    schedule_name   => 'hr.employee_dept_chg_schedule',
    enabled         => FALSE
  /* The parallel_instances attribute says that this job should start a new
   * instance for each message on the queue, even if an instance is already
   * running. */
  DBMS_SCHEDULER.set_attribute ('hr.employee_dept_chg_job',
  DBMS_SCHEDULER.enable (name => 'hr.employee_dept_chg_job');

As you can see, this is quite a bit more involved than simply using DBMS_JOB.  But perhaps you are doing something more involved than simply sending an e-mail?  If so, AQ and SCHEDULER are a very powerful combination to get things to happen automatically when the database changes, but ONLY when the change is saved.