21-August-2003
Craig Russell
Attendees: David Tinker (JDO Genie), Markus Kuefer (SAP), Michael Bouschen (Tech@spree), David Ezzio, Eric Samson (LIBeLIS/LiDo), Dirk Theune (POET/FastObjects), David Jordan, Pinaki Poddar (Versant/enJin), Wes Biggs (XORM), Guy Hillyer (Progress/ObjectStore), Sumanth Kommareddy (Progress/ObjectStore), Abe White (SolarMetric/Kodo JDO), Neelan Choksi (SolarMetric/Kodo JDO), Patrick Linskey (SolarMetric/Kodo JDO), Dennis Leung (Oracle/TopLink), Mike Keith (Oracle/TopLink), Robin Roos, Matthew Adams, Dion Almaer (TheServerSide), Craig Russell (Sun).
Tuesday 12-Aug-2003
The meeting started with introductions by all parties. The agenda was agreed, although in hindsight parts of the agenda were allocated more time than necessary, and parts were hopelessly underestimated.
It was announced that since the JSR for JDO 2.0 had not been submitted nor approved by the JCP EC, this meeting would be held in the open. Nothing discussed in the meeting would be proprietary or covered by non-disclosure. So anyone is free to openly discuss the results of the meeting in public. We all agreed to the policy of the meeting.
AI Craig Russell: Once the JSR is approved, when can vendors release products that conform to the draft specification without running afoul of the JCP rules?
[formal answer: read the JCP JSR process. informal answer: once the JSR is approved by the expert group for public review, vendors are free to ship pre-release software that implements the spec, as long as the pre-release software isn't final and can't be used in production.]
The draft JSR was amended to tighten the high level objectives and to reword the section on J2SE support. We want to take advantage of J2SE 1.5 features for ease of use while retaining J2SE 1.3 compatibility.
We agreed that JDO 2.0 would use xml schema to represent the format of the xml metadata, but for the purposes of this meeting and for preliminary communication among members, we would use the DTD format. Partly this is because of the lack of expertise in the group of xml schema and partly because the DTD format is easier for humans to parse.
We agreed that the Java Data Object-Relational specification would address at least two different areas: specification of the metadata to represent the mapping of jdo classes and interfaces to relational databases; and specification of the runtime behavior of classes so mapped.
The details of the mapping were delegated to a working group headed by Patrick Linskey, but we agreed on the objectives:
AI Robin Roos: Provide the JDO 2.0 working group details of the JDOR metadata.
The draft JSR was amended to tighten the high level objectives and to reword the section on J2SE support. We want to take advantage of J2SE 1.5 features for ease of use while retaining J2SE 1.3 compatibility.
Craig Russell presented the objective of expanding the scope of JDO persistent types to include interfaces. The <interface> element would be added to the JDO metadata parallel to the <class> element. Bean patterns would be used to identify default persistent fields, so interfaces could be made persistent without a lot of work on the part of the developer, and less work on the part of the developer would be needed when adding a persistent virtual field. That is, once the interface was updated to add a new get/set pattern, no change would be needed in the JDO metadata to make the new virtual field persistent.
This would allow users to write their application domain data models strictly by interfaces, instead of by concrete classes as JDO 1.0 currently requires. We might extend this to also allow support for abstract classes that have no implementing classes provided by users.
AI Craig Russell: Write up the proposal for persistent interfaces.
Robin Roos presented his “top ten” proposals for JDO 2.0. Many of these had been sent out as meeting preparation materials, but Robin presented a subset of proposals for immediate adoption.
AI Robin Roos: Write up the adopted/deferred proposals.
AI Craig Russell: Write up the F06 EventListener proposal.
We discussed Dirk Theune's short list:
Nested transactions. We agreed that single-nested transactions can be defined that have a simple semantic operating only on the cache. Commit of a nested transaction propagates any changes to the parent transaction in the cache only. Rollback of a nested transaction cancels changes to the cache. Only the top transaction commit will change the database. Flushing the cache in case of queries when a nested transactions is active was discussed but not resolved. We discussed savepoints and checkpoints as possible implementations of nested transactions or as stand-alone features. Deferred.
OQL. Currently vendors who support OQL can do so via the PersistenceManager.newQuery(String, Object) interface method. But the way to return results is not standard. Standardizing the behavior of OQL queries can be seen by the community as JDO endorsing object databases. We agreed to have standard behavior for OQL only in conjunction with standardizing behavior of SQL queries as well. Deferred until after JDOQL extensions discussion.
Index specification in JDO metadata. Standardize how to specify indexes for schema creation in metadata. This is not a JDOR feature, as all datastores have the concept of indexes to improve performance. Adopted.
SCO embedded semantics. We need to define the semantics of embedded fields. This is left over from the discussion in JDO 1.0 of how to mandate the behavior of embedded fields of PersistenceCapable types. Adopted.
AI Dirk Theune: Write up proposals.
We discussed the recovery from a failed optimistic transaction. Currently the spec requires that instances that failed the check would have to be refreshed, but this is a bit of work. The user has to iterate the exception, analyze what failed, and refresh the instances. It would be nice to have a convenience method to refresh all failed instances in preparation for retrying the operation. PersistenceManager.refresh(JDOOptimisticVerificationException);
This might be not quite useful enough, though. It might be better to provide another way to obtain the list of failed instances without needing to commit to get it. PersistenceManager.attemptSynchronize() which would convert the optimistic transaction to a datastore transaction and attempt to synchronize the state of cached instances. The failed instances would be reported in an exception similar to applyDetached. The user could modify the instances and retry the updates.
AI Craig Russell: Write up proposal.
We discussed explicit locking of instances, which generated a lot of heat but little light. Some experts really want it; others are really opposed to it, as they think it represents bad practice and exacerbates portability issues. Without much hope that we can reach agreement any time soon, we deferred the discussion to email.
AI Pinaki Poddar: Write up proposal.
Matthew Adams proposed trying to avoid the requirement for a no-arg constructor. The idea is that the no-arg constructor might have application semantics, and using it for JDO violates transparency. The metadata could have an element that would be used by the JDO implementation instead of the no-arg constructor. The element would describe a static method of the class or a method of a factory TBD.
AI Matthew Adams: Write up proposal.
We concluded Tuesday's meeting by deferring discussion of details of inheritance in JDOR metadata, managed relationships, and JDOR SQL integration.
Wednesday 13-Aug-2003
We reviewed the high level objectives for disconnected operation and discussed two use cases in some detail:
The requirements are to specify the semantics of disconnecting persistent instances from the graph of cached instances and the semantics of applying changes to corresponding persistent instances from a different PersistenceManager (although the use case of the same PersistenceManager with the same disconnected instances must be supported as a special case). The disconnect must operate in a way that guarantees that all of a specified set of fields in the disconnected instances are valid, and the other persistent fields have their Java default values. The reconnect must provide for the application to force changes to be applied under application control (a report of the failed attempt must allow the application to handle the merge).
We agreed that the disconnect operation would make a copy of the instances, and the reconnect would similarly apply changes to instances from the PersistenceManager's cache, and not attempt to make the reconnected objects part of the cache. We agreed to revisit this decision if there are significant performance or usability issues associated with this strategy.
The disconnected instances contain their object id and version id in a form that can be transmitted across VM boundaries. Thus there must be a way to specify the object id and version id fields in the metadata and a way to get the version id from a persistent instance (there is already a way to get the object id). There must also be a way to get the object id and version id from the disconnected instance. We left open the question of whether the application can get the object id and version id in a standard way from the instance once it's outside the original context (VM and PersistenceManager).
Detached instances form a closed set of instances, as if the instances were serialized to an external form and deserialized to an internal form. The detach API should return a collection of copies of instances so the application can associate them with the collection of persistent instances. Dirty instances should be flushed prior to making the copy so that version information is updated. If the transaction in which the flush was performed fails, then the result of the disconnect must be discarded (otherwise data corruption might occur).
For each class of instance involved in the disconnect, the user must define a set of fields to include and exclude (by definition, all non-included fields are excluded). This definition is an unresolved detail, but it is expected that a fetch group concept can be used for this purpose. The disconnected instances must have all of their included fields valid, and Java default values in all excluded fields. It might be useful to guarantee that the application cannot even access excluded fields, so we might consider having the PersistenceManager throw an exception if an excluded field is accessed from a disconnected instance. This implies that the disconnected instances are minimally managed.
The disconnected instances might be sent to the client directly as value objects (just as instances can be sent by value after makeTransient in JDO 1.0.1). We will consider deprecating makeTransient as no one (including its author) likes it at all.
The identity of disconnected instances must be available to the application. We will consider adding an API to PersistenceManager to return the version id of a disconnected instance. We will consider adding an interface or class Handle to hold the object id and version id of disconnected instances.
AI David Ezzio: Write up proposal to access version id without adding a Handle class/interface.
package javax.jdo;
class Handle {
Object oid;
Object vid;
Object getOid() {
return oid;
}
Object getVid() {
return vid;
}
Handle (Object oid, Object vid) {this.oid = oid; this.vid = vid;}
}
Handle PersistenceManager.getHandle(Object);
Handle JDOHelper.getHandle(Object);
Object PersistenceManager.getObjectById(Handle);
Object PersistenceManager.getObjectById(Handle, boolean);
AI Craig Russell: Write up proposal for disconnect/reconnect.
We reviewed the list of Exceptions that should have standard behavior and added several to the list of standard Extensions. These Exceptions have analogs in all datastores and should be required to be thrown for specific conditions. The Exceptions to be standardized are:
JDODuplicateObjectIdException, JDOClassNotPersistenceCapableException, JDOExtentNotManagedException, JDOConcurrentModificationException, JDOQueryException, JDOQuerySyntaxException, JDOUnboundQueryParameterException, JDOTransactionActiveException, JDOTransactionNotActiveException,JDODeletedObjectFieldAccessedException, JDODeadlockDetectedException, JDOLockNotGrantedException, JDOPostCommitException, and JDOPostRollbackException.
AI Craig Russell: Write up proposal for new Exceptions.
We agreed to consider adding a standard API to invoke a byte-code enhancer, but agreed that it is lower priority than most other items on our agenda.
AI Craig Russell: Write up proposal for enhancer API.
We agreed that it is useful to mark persistence-aware classes in the JDO metadata, so that a clever tool can know all classes that need to be enhanced.
AI Craig Russell: Write up proposal for persistence-aware classes in metadata.
We discussed the validate parameter to getObjectById and concluded that it is confusing. Apocryphally some users on the same project always use true, and others always use false. Since the 90% case is true, we should have a new API in which the parameter is omitted and assumed to be true: void getObjectById(Object);
We also agreed that we need methods that allow multiple objects to be loaded in order to take advantage of group operations: Object[] getObjectsById(Object[], boolean); Collection getObjectsById(Collection, boolean); Object[] getObjectsById(Object[]); Collection getObjectsById(Collection);
AI Craig Russell: Write up proposal for getObjectsById and getObjectById .
Thursday 14-Aug-2003
Query extensions were presented by David Ezzio and Michael Bouschen.
Projections and aggregates extend the results of a query to include data that is not simply a collection of candidate instances. We agreed to use Query.setResult(String) to represent the user's request for projections or aggregates.
We need a better way to analyze the result of a query, and make it easier to specify that there is a single result instance.
We added an API Query.setUnique(boolean) to indicate that there is a single result instance.
We added an API Query.setLimit(int number) to indicate that only the first number of instances are to be returned, and the rest discarded. We briefly discussed paging query results (getting "the middle" results from a large query) but did not come to a conclusion.
We added an API Query.setResultType(Class userResult) to specify the class of instances to be returned by a projection query.
We realized that to be completely useful, aggregates need grouping. The simple grouping is not general enough. So we added an API Query.setGrouping(String grouping) to specify the columns by which to group.
|
setResultType |
setResult |
setUnique |
setLimit |
Shape of result of execute() C is Candidate class |
|---|---|---|---|---|
|
ignored |
null |
false |
0 |
Collection<C> |
|
ignored |
null |
false |
n |
Collection<C> size<n |
|
ignored |
null |
true |
n/a |
C |
|
no |
!null |
false |
0 |
Collection<Object[]> |
|
no |
!null |
false |
n |
Collection<Object[]> size<n |
|
no |
!null |
true |
n/a |
Object[] + |
|
UserResult + |
!null |
false |
0 |
Collection<UserResult> |
|
UserResult |
!null |
false |
n |
Collection<UserResult> size<n |
|
UserResult |
!null |
true |
n/a |
UserResult |
+ Note 1: TBD Object[] might be replaced by a single instance of a specific type for single projections/aggregates.
+ Note 2: UserResult is a user-defined class, not a javax.jdo interface or class.
We will define aggregates to include: min, max, count, sum, and avg, and define the Java types returned for each aggregate type. We left unresolved whether these are keywords in the JDOQL language or need to be qualified, e.g. Query.min. There is a desire to keep the query as close to Java as possible, but keywords are easier to use.
The UserResult class is only for projections and aggregates, and is ignored for cases where setResult is null or default.
For each element in the result row, the UserResult class is introspected and one of four cases is executed:
We agreed to support interfaces in some specific cases. If an interface appears in the metadata, an extent can be managed for the interface. In this case, querying on the extent of an interface will use the union of all classes that implement that interface as the candidate collection. Fields that can be queried are virtual fields that are mapped via metadata to fields in each class that implements the interface. The mapping is specified in each class.
Navigation through interfaces is a bit trickier, and we agreed to support a very limited case where there is exactly one persistent class that implements that interface. This restriction will provide for portable applications. A JDO implementation can relax this restriction but any application that uses this will not be portable.
We discussed the overloading of execute in light of delete, unique, and J2SE 1.5 which will allow specification of the return type as part of the signature. One possibility is to add translation APIs to Query which allow users to convert the overloaded execute parameters to a single type Object[] which is then the only required overloaded method.
Object[] convertParameters(Object o);
Object[] convertParameters(Object o1, Object o2);
Object[] convertParameters(Object o1, Object o2, Object o3);
Object[] convertParameters(Map m);
With the above, we can have:
Object delete(); // the no-parameter version Object delete(Object[]); // the only parameter version Object delete(Object...); the J2SE 1.5 way, identical to Object[]
A different technique is to define new methods to set the parameters: void setParameters(Object o); void setParameters(Object o1, Object o2); void setParameters(Object o1, Object o2, Object o3); void setParameters(Map m);
The issue with these is that the Query instance is now bound to parameters, which is not thread safe any more. Since a very common use case is to define a Query once and use it multiple times with different parameters (e.g. findByPrimaryKey) this technique might not be optimal.
We discussed allowing a standard way to specify non-standard Query behavior. Query.setProperties(Properties) and Query.setProperty(Object, Object); There would not be any standard behavior for these, but a vendor could use them to specify modifications to the standard Query behavior.
AI Patrick Linskey: Write up setProperty and setProperties.
New String methods are supported by JDOQL:
String toLowerCase(), String toUpperCase(), int indexOf(String), int indexOf(String, int), boolean matches(String), String substring(int), String substring(int, int).
These methods all appear in java.lang.String, so there is no loss of mapping to the language. But to allow for implementations on relational databases, the String passed to matches is a subset of regular expressions that includes only (?i) for case-insensitive matches and . and .* for wild card matches. If an implementation chooses to support more of the regular expression syntax, this will not be portable. The EJBQL LIKE and LIKE...ESCAPE functions can be implemented with matches.
We will use 0-based indexes unlike EJBQL which uses 1-based indexes.
Two Math functions and an operator are supported for filter expressions: Math.sqrt (square root), % (modulo), and Math.abs (absolute value). Vendors can add other Math.functions, but these are not portable.
We allow queries on mapped Maps, to allow containsKey(Object), containsValue(Object), and isEmpty().
We support the distinct keyword on setResult to specify removal of duplicate rows of the return result.
We support an instanceof test to filter results based on the mapped type of candidates, fields, and variables.
We discussed named queries, which is the ability to define a query in an external form, such as a properties file or in the metadata. We agreed that this is a useful addition and eagerly await the proposal.
AI David Ezzio and Michael Bouschen: Write up query proposal.
Guy Hillyer and Sumanth Kommareddy presented a proposal to add a Placement concept to the API. This notion has been implemented in ObjectStore for several years, and allows the application developer better control over exactly where objects are stored. There is an analog in relational mapping products where several databases are logically merged so one PersistenceManager is used, but the instances retain their association with a specific part of the database.
AI Guy Hillyer: Write up proposal.
Eric Samson presented a UseCase scenario to allow the application to specify a set of application-specific behaviors, including the fetch groups for each application class, concurrency mode and isolation level for the transaction, and locking behavior.
AI Eric Samson: Write up use case proposal.
We agreed to try to specify a mix of SQL and JDO query, where the result of a query is a collection of JDO instances. This caters for the DBA who wants to specify the exact SQL to be used for a query but the application wants to use only JDO instances not ResultSets.
We agreed to specify a standard behavior for PersistenceManager.newQuery("javax.jdo.SQL", <SQL Query>). The SQL Query would be restricted to one of SELECT, WHERE, and CALL. Given some rules (the user would have to specify the candidate class via setClass, the key columns and version number columns would have to be in the SELECT clause or returned in the ResultSet) the result of the query execute would be a Collection of persistent instances that could be used with the JDO API. No requirement to parse or analyze ResultSet. If the user really wants ResultSet, she can get a Connection and go wild.
AI Eric Samson: Write up proposal.
Binary compatibility: We discussed the possibility of giving up on binary compatibility. The issue is that it appears to have very specific use cases and it is a hindrance to implementing a reflection-based JDO. The original use case is for application servers where persistent classes would be loaded by the system class loader and used by many applications. We have since come to understand that applications are loaded by their own class loader, so the same class would be loaded with a different Class instance by each application. The only drawback of losing binary compatibility in this context is for service classes that are loaded by the system class loader.
Reflection-based JDO implementation: Mike Keith presented several areas of the current specification that are difficult or impossible to implement using a reflection based approach. These include: the requirement to report the status of instances (persistent, new, deleted, transactional, and dirty); and navigation through a lazily-loaded single-valued persistent object reference.
The sense of the meeting is that we should look at these issues and see how far we can change the specification to accommodate reflection-based solutions without compromising the transparency that JDO users have come to expect.
Managed Relationships: We discussed the semantics of managed relationships and agreed that it was not just an object-relational concept, but an object modeling concept. The concept is that a relationship is represented by a field on both sides of the relationship plus behavior that updates both sides if one side is changed by application code. We could add metadata to describe this relationship and generate code to implement it without regard for whether the instances are persistent or which type of datastore is in use. The implementation might depend on what type the reference is (List v. Set). We came to no conclusion as to whether this is implementable, but we will take it up later.
Expert Group communication vehicle: We agreed that the current email communication strategy is inadequate, but is all that the Specification Lead can sign up for. Other ideas were a threaded discussion group, a wiki, and a newsgroup. Any of these will need a host.
AI Patrick Linskey: Set up a discussion group.
Working Groups were assigned team leads. The idea is that all discussions will be held on the official JDO discussion forum and the team lead is responsible for driving the discussion to conclusion.
JDOR schema: Patrick Linskey
Disconnect/reconnect: Craig Russell
Generics: Wes Biggs
OID/Version id: David Ezzio
Query: Michael Bouschen
Fetch groups: David Tinker
Use Case (Policy): Eric Samson
Exceptions: Craig Russell
Callbacks, events: Robin Roos
Persistent Garbage Collector: Matthew Adams
Extent of an interface: Robin Roos
Load all metadata: Patrick Linskey