Thursday, May 30, 2013

Fix: Endeca Baseline fails with NullPointerException in pushAuthoringContentToDgraphById

Symptom 1: When running a Baseline Index in Endeca 3.1.1, the MDEX server(s) do appear to be updated, but several errors appear in the console output, the first of which is:

[05.29.13 14:06:10] INFO: Pushing authoring content to dgraph: AuthoringDgraph
[05.29.13 14:06:10] SEVERE: null
Occurred while executing line 5 of valid BeanShell script:

3| compId = invokingObject.getElementId();
4|"Publishing Workbench 'authoring' configuration to MDEX '" + compId + "'");
5| IFCR.pushAuthoringContentToDgraphById(compId);

[05.29.13 14:06:10] WARNING: Caught an exception while executing post-startup script for server component 'AuthoringDgraph'.

Caused by com.endeca.soleng.eac.toolkit.exception.AppControlException
com.endeca.soleng.eac.toolkit.script.Script runBeanShellScript - Error executing valid BeanShell script.
Caused by java.lang.NullPointerException
com.endeca.soleng.eac.toolkit.utility.IFCRUtility extractIfcrErrorMessage – null

Symptom 2: Running the control/ script fails with a similar error.

Symptom 3: Trying to add a new content item in the Endeca Workbench Rule Manager fails with the error “Last attempt to publish to the MDEX Engine failed”.  If the mouse cursor is hovered over the error message, a tooltip appears with an additional error message: “com.endeca.ifcr.publish.impl.PublishException: Failed to publish to the following dgraphs: [dgraph url]”.

Solution: This started happening for my new Endeca application after I changed the password for the Endeca Workbench “admin” user to something other than the default value of “admin”. 

Setting the correct (updated) password for the admin user in the application’s config/script/WorkbenchConfig.xml file resolved the issue.

Credit to Dan O. from Thanx Media for getting this issue tracked down.  Thanks Dan!

Tuesday, May 28, 2013

Oracle ATG – Endeca integration: Baseline Index process troubleshooting

This article contains troubleshooting steps for when an Endeca Baseline Index, as manually launched via the ATG Dyn Admin | ProductCatalogSimpleIndexingAdmin component, doesn't complete successfully.

All of this information is from my own experience and troubleshooting in trying to get an ATG-Endeca integration for a new ATG/Endeca application wired up and working successfully. (Which I did eventually accomplish, but not until after working through a fair number of unexpected issues, as should be apparent from the list below!) 

So, disclaimer: The suggested resolutions may or may not be “best practice” or otherwise work well for your own application. I’m just posting this information in hopes that it may be helpful; I certainly wished at points that there was more ATG-Endeca integration help available on the web at times, while I myself was initially troubleshooting my way through each one of these issues!

Q: The ProductCatalogSimpleIndexingAdmin doesn't exist in Dyn Admin. Searching for it returns 0 results. Browsing for it shows that the /atg/endeca subtree doesn't exist.

1. The "ATG-Endeca integration" feature might not be installed in ATG. This can be done with the atgdir/home/bin/ utility.

2. The application startup might not have included the required Nucleus modules for the Endeca indexing integration. In my situation, I found that I needed to add the following Make sure the following Nucleus components are included with the application server startup – these may or may not be the specific set of modules that are appropriate for your ATG application:

  • DAF.Endeca.Index
  • DCS.Endeca.Index

Q: In Dyn Admin, the ProductCatalogSimpleIndexingAdmin component does come up, but the special UI that normally appears at the top of that component's Dyn Admin page, that allows a Baseline Index to be launched and shows the status of the process, doesn't appear at all.

Try inspecting the application startup log for errors and warnings.  I saw this happen when my product-sku-output-config.xml file contained an extra “-->” token; this was reported as a parse error in the application startup logs.

Q: Errors about missing tables appear in the log, including table srch_site_content, both at server startup and when trying to run Baseline Index.

I ended up needing to manually run some ddl files in my local dev environment to resolve this. I ran:

On core:

  • search_site_ddl.sql

On cata and catb:

  • versioned_search_site_ddl.sql

These scripts are packaged with ATG. Look for them in one of the descendant directories of the ATG_HOME directory.

Q: Under JBoss, trying to run Baseline Index via ProductCatalogSimpleIndexingAdmin fails. Exception appears in the log: " java.lang.NoSuchMethodError: org.apache.cxf.transport.AbstractTransportFactory.<init>(Ljava/util/List;)"

This is apparently due to a version conflict between the version of CXF that comes packaged with JBoss, and the version of CXF that ATG assumes will be available to it.

I spent some time looking into how to disable or otherwise not use the CXF version bundled with JBoss (and use ATG's version of CXF instead), but never found a solution. I ended up switching my local environment to use WebLogic instead or JBoss as a workaround.

Here's a post on the Oracle forums with this same problem (and no solution, as of 5/2013):

Q: The second portion of the Baseline Index process (RepositoryExport) fails immediately.

Possible additional symptom in log: /atg/commerce/endeca/index/ProductCatalogSimpleIndexingAdmin ---java.lang.RuntimeException: org.apache.commons.httpclient.ConnectTimeoutException: The host did not accept the connection within timeout of 30000 ms

There might be a problem with the connectivity between ATG and Endeca CAS – possibly a configuration issue or networking issue. Check:

1. In the DimensionDocumentSubmitter, SchemaDocumentSubmitter, and DataDocumentSubmitter components, are the CASHostName and CASPort properties set properly? If not, make the changes in the corresponding .properties files for each of those components (in ATG 10.1.1), or in the /atg/endeca/index/IndexingApplicationConfiguration/ component (in ATG 10.1.2).

2. Try pinging the target Endeca CAS server from the command line on the ATG machine to verify that it is accessible on the network. 

3. Verify that DNS resolution isn’t misconfigured such that the CAS hostname is resolving to an IP address of some server other than the actual CAS server.

Q: The second portion of the Baseline Index process (RepositoryExport) fails immediately. The log includes "Response-Code: 500"

The "500" here appears to be an HTTP 500 ("Internal server error") being returned from Endeca CAS from the web service call. I've seen this happen several times when attempting to run a Baseline Update after a previous Baseline Update failed midway through, and when the indexing process is being run from the ATG CA server.

The only workaround I know of at this point is to bounce the CAS server. Quick instructions to bounce CAS (in my Endeca environment; specifics for your environment may vary):

  1. ssh to the Endeca indexing server.
  2. Change directory to endeca/CAS/version number/bin
  3. ./
  4. nohup ./ &

Q: The second portion of the Baseline Index process (RepositoryExport) hangs for a while, and then fails with no records successfully processed. An error related to network connectivity appears in the ATG server log.

1. Is CAS running? Log on to the Endeca indexing server and run:

ps -ef | grep cas

If there's no result, then start CAS.

2. Alternatively, this might indicate a DNS issue. Make sure that when you ping the Endeca indexing server from your ATG server, the correct IP address is being returned.

Q: http://[endeca host]/[endeca app name]en_en_dimvals/?wsdl returned response code 404

This appears to be caused by an incorrect defaultRecordStoreName and/or endecaBaseApplicationName entry in the DataDocumentSubmitter, DimensionDocumentSubmitter, and/or SchemaDocumentSubmitter components (ATG 10.1.1) or in the IndexingApplicationConfiguration component (ATG 10.1.2). Check on those values for those components in Dyn Admin and/or in the application’s .properties files.

One specific thing to check: If the language code (e.g. “en”) is being specified in the endecaBaseApplicationName property, this might result in a defaultRecordStoreName value with too many language code values, e.g. “MyAppenen_en_dimvals”, which isn't correct – too many "en".

Q: Exception in thread "index-/atg/endeca/index/commerce/ProductCatalogSimpleIndexingAdmin" java.lang.NullPointerException: Property value cannot be null (dimval.display_name)

This apparently indicates that one or more of the item-descriptor tags in customCatalog.xml (in the ATG project) are missing display-name attributes. (Note that some item-descriptor tags – particularly, the OOTB ones – inherit a display-name via the xml-combine="append" attribute value.)

Another symptom of this: In Dyn Admin, RepositoryTypeDimensionExporter component, under "show XML", one of the dimval.display_name entries in the XML shown has an empty PVAL.

Q: The ProductCatalogOutputConfig step reports success, but shows 0 records processed. The server log doesn't show any warnings.

Additional symptom: The Dyn Admin | Commerce Admin | Catalog Verification reports thousands of warnings like: "/atg/commerce/catalog/custom/CatalogVerificationService Product 434318 is not in any catalog. Cannot verify info objects."

Additional symptom: When looking the properties for any particular product via the ProductCatalog component in dyn admin, several properties are missing: ancestorCategories, ancestorCategoryIds, siteIds, parentCategoriesForCatalog, catalogs, computedCatalogs.

To fix:

  • In Dyn Admin | Commerce Admin, run Catalog Update and then Basic Maintenance. Then, run Catalog Verification; it should report 0 errors, 0 warnings.
  • If an error occurs during Basic Maintenance, it might be caused by an database deadlock. (You can look at the application log file to verify whether or not this is the case.) 

Q: ProductCatalogOutputConfig step fails after around 200 records processed. Application log shows: SQLException: Internal error: Cannot obtain XAConnection weblogic.common.resourcepool.ResourceLimitException: No resources currently available in pool

This apparently indicated that ATG was trying to use more connections than were available in WebLogic. I’m not sure what caused this when I saw it.  I ended up just retrying the baseline index (without changing any settings or bouncing the server), and it worked the second time.

Q: java.lang.OutOfMemoryError occurs after processing some, but not all, of the product records.

I worked around this by increasing the Java max heap size (for the ATG application) to 3G, and the MaxPermSize to 384m.

Q: Silent failure on the EndecaIndexing step (the last step). Nothing appears in the ATG application log after the RepositoryExport steps completed. The Dyn Admin GUI just reports "COMPLETE (Failed)".

1. This may indicate that the forge process failed on the Endeca side.  Try looking at the logs on the Endeca indexing server for details about the failure.

2. Verify that EAC is running on the Endeca indexing server.

3. Verify that in the EndecaScriptService (ATG 10.1.1) or ApplicationConfiguration (ATG 10.1.2) component, the endecaBaseApplicationName (10.1.1) or baseApplicationName (10.1.2) property has the correct value.

Q: Second portion of baseline index (RepositoryExport) fails midway through. Error in log: "/atg/search/repository/BulkLoader — com.endeca.itl.recordstore.ConcurrentWriteException: Write in progress with generation 12"

I found that a workaround for this was to restart CAS on the Endeca indexing server, then retry the baseline import from the ATG side.

Update 8/19/2013: Apparently this issue can be caused by an update lock not being released when a previous baseline index operation was manually canceled. See article "Cancelling the Baseline Indexing Job from SimpleIndexingAdmin does not Release Update Lock (Doc ID 1576472.1)" on the Oracle Support site for more information.

Thursday, May 09, 2013

Workaround: Mac Mini outputs no video signal with DisplayPort-DVI adapter (and some amazing customer service)

Executive Summary / tl;dr

  • A Mac Mini might produce no video output when connected to a ViewSonic or Dell monitor via a DVI-DisplayPort adapter.  Try using a DVI-MiniDVI adapter instead.
  • For incredible customer service, buy Mac stuff from eBay seller dvicetech.

Although I’m a longtime Windows user when it comes to desktop PCs, over the past couple of years I have developed a strong liking of Apple’s iPod Touch as my “always-in-my-pocket” device of choice (as an inexpensive smartphone alternative).  With 3 young kids in the house, my free gaming time is increasingly spent on the iPod, as that platform is conducive to 10-15 minute play sessions; however, the particular game I’d like to play is not currently on the App Store, so being a programmer, I decided to write it myself!  (Hopefully, more to come on that topic in a future post on this blog!)

Being brand new to iOS development, I did some research, and ended up with the conclusion that unfortunately, using my existing Windows PC would not be a very good idea; Apple really wants you to be using a Mac for iOS development.  Being on a fairly constrained budget for discretionary purchases – again, see “3 young kids” above! – I opted for the “Mac Mini from eBay” option, as suggested on StackOverflow.

After examining the available options, and (as always with this type of purchase) looking for a sweet spot in the trade-off between price and performance, I opted for a so-called “Early 2009” model Mac Mini, with 2.0 GHz dual-core processor, 4 GB RAM, 500 GB HD, and the as-of-now latest OS, OS X 10.8 Mountain Lion, from eBay seller dvicetech.  This Mac Mini cost $444.50, including shipping; using my existing keyboard, mouse, and monitor, that was the total cost for the Mac.  (Definitely a more economical option than going with a new MacBook, which seem go in the $1100 – 1700 range.)

When my Mac Mini arrived, I hooked it up, connecting a USB mouse, USB keyboard, power, and my existing DVI monitor via a DisplayPort-DVI adapter included by the seller.  I turned the machine on, at which point the “power” light on the front of the Mac Mini turned on, I heard the Apple startup chime from the Mac’s built-in speakers, and… nothing at all appeared on the screen.  The monitor’s status indicator light remained yellow, as though no input source were connected.

This monitor, a ViewSonic, did work fine with my Windows PC; I had just unhooked it from that PC in order to use it with the Mac.

I checked the standard set of things that I could think of to check:

  • Monitor powered on?  Yes, the yellow status indicator light was on.
  • Mac powered on?  Yes, the power indicator LED was on and I heard the startup chime.  I could also toggle Caps Lock on and off via the keyboard and see the Caps Lock indicator light on the keyboard turn on and off.
  • Video cables connected tightly to Mac and to monitor?  Yes.
  • Monitor configured for DVI input?  Yes, verified.  I’d also been using this monitor with my Windows PC via DVI with no problems.
  • DVI cable okay?  I swapped it out for a different DVI cable that I had on hand, but got the same results.

At this point, I disconnected the ViewSonic monitor, connected my other on-hand monitor (a Dell), and rebooted the Mac Mini.  This also produced the same results: Mac Mini appeared to power up okay, but no display output whatsoever.

So at this point, I concluded that the problem was either with the DisplayPort-DVI adapter, or with the Mac Mini itself.  I Googled using some general terms (“mac mini no video”), and came up with a Apple support article for this specific problem, which had several suggested solutions.  Unfortunately, when I tried the solutions, including several new-to-me troubleshooting actions like “reset the parameter RAM” and “reset the System Management Controller (SMC)”, none of them resolved the problem.  (Also, I was unable to try at least one of the listed solutions, which was to put the Mac to sleep by pressing Option+Command+Eject; the PC keyboard I was using with the Mac Mini in accordance with the Mac Mini’s advertised B.Y.O.D.K.M. feature didn’t have any of those keys!)

Having run out of troubleshooting steps to try, I contacted the eBay seller, dvicetech, through eBay’s nice dispute resolution site feature and described the problem, in hopes of being able to either get help in resolving the problem somehow (although my hopes for that were limited at this point), or else exchange the Mac Mini for another one, or get a refund. 

The seller, dvicetech, immediately suggested shipping me out a replacement DisplayPort-DVI adapter, at his own expense. This really pleased me, since combined with dvicetech taking on this extra cost, it seemed like a very reasonable next step to try – hopefully the problem would turn out to be with the adapter, not the Mac Mini itself. 

Unfortunately, when the new DisplayPort-DVI adapter arrived and I hooked it up, it didn’t resolve the issue. 

As a last-ditch troubleshooting attempt before I contacted dvicetech again with the bad news and to ask for either a replacement or a refund – I was leaning toward the latter, as at least then I’d have the option of spending an additional $200 or so on a new Mac Mini and engaging Apple Support if something were to go wrong with that – I brought the Mac Mini and the DisplayPort-DVI adapter into the office to try it with the monitors there, on the wild theory that there was some problem with both the ViewSonic and Dell monitors I had at home where those monitors worked fine with Windows 7, but didn’t work with the Mac. 

To my shock, this worked!  When I connected the Mac Mini to the Samsung DVI monitor that I had at work – using the same Mac Mini and DVI-DisplayPort adapter that had previously failed to produce any video signal on either the ViewSonic or Dell monitors that I had at home – I did get signal on the Samsung monitor.  I was able to watch the Mac Mini boot to the MacOS desktop with no issues.

I contacted dvicetech with this news.  At this point, dvicetech wowed me further with his great service by shipping out not one but two additional display adapters to try: A miniDVI-DVI adapter, and a miniDVI-VGA adapter – along with a postage-paid return envelope for whatever display adapters I didn’t end up using!  I hadn’t even realized that using something other than a DisplayPort adapter was even an option.

When this third round of display adapters arrived, I connected the Mac Mini to my home office ViewSonic monitor via the new miniDVI-DVI adapter, and bam! – instant video signal!  The Mac Mini booted up and worked just fine at that point. 

So I put sent the three remaining, unused display display adapters back to dvicetech using the envelope that he had provided.  I was seriously blown away by the customer service dvicetech provided – excellent, prompt, polite communication; smart troubleshooting suggestions; and multiple packages shipped back and forth at his own expense – which I daresay beat out even what I would have gotten from Apple directly, had I purchased a new Mac Mini.  At a minimum, I would have been spending $5+ in gas money per round trip to the nearest Apple store, which is about 30 minutes from my home.  My advice is when shopping for parts on eBay, buy from dvicetech if possible – I certainly intend to do so in the future!

I’m somewhat less happy with Apple, with all of the time and hassle it took me to get the new Mac Mini working.  When I use some more specific Google search terms like mac mini viewsonic no video, there are lots of other similar problem reports.  This wasn’t exactly an “it just works!” experience.  My takeaways are:

  • For Mac video adapters, MiniDVI-DVI is a more reliable option than DisplayPort-DVI, although limited sample size is certainly a caveat here;
  • Don’t assume everything will be instant sunshine and roses with Apple hardware, especially when interoperating between Apple hardware and standard-but-non-Apple hardware (although again, sample size caveats apply).

Still, in the end, the Mac Mini is now up and running, and I’m well along in coding up my new game in Xcode, Apple’s IDE for iOS.  It’s turning out to be a big project, so I’m several months away from even being ready to beta the app (particularly since I’m almost exclusively working in 90-minute-per-day chunks at the end of the evening after my kids are all in bed and other chores are all done), but I’ll certainly make an announcement here when I have something ready to show!