ATG BCC – “Unknown Segment ‘Main’” log error; missing menu items in left nav

This post details a problem, and its corresponding solution, that my team encountered and recently resolved in the ATG 9.3 BCC application.

Symptoms

1. In the BCC application, in the left nav menu, several of the menu items that should be present are missing, including:
   • Content Administration | Automated Deploy
   • Content Administration | Content Administration Project
   • Merchandising | Manage Commerce Assets
   • Personalization | Targeting and Segmentation
     
2. When a user successfully logs into the BCC application, lots of errors like the following are logged in the CA application’s log file: atg.process.ProcessException: Unknown segment "main" in process /Content Administration/CA.wdl at atg.process.ProcessManagerService.getSegmentInfo(ProcessManagerService.java:2122) at atg.process.ProcessManagerService.createProcessInstanceInfo(ProcessManagerService.java:11085) at atg.process.ProcessManagerService.getProcessInstanceInfos(ProcessManagerService.java:10928) ...
3. When the CA instance is initially started, some errors like the following that mention “atg.nucleus.ServiceException” and “dynamoMessagingSystem.xml” appear in the CA application’s log file: atg.nucleus.ServiceException: Destination "sqldms:/Price/CacheInvalidation" is not found for output port "PriceCacheInvalidator" of "message-source" element with Nucleus name "/com/example/ecom/messaging/PriceCacheMessageSource" for patch bay definition file "/atg/dynamo/messaging/dynamoMessagingSystem.xml" at atg.dms.patchbay.PatchBayManager.createOutputDestination(PatchBayManager.java:1451) at atg.dms.patchbay.PatchBayManager.createOutputPorts(PatchBayManager.java:1249) ...

Solution

(This solution worked for my team’s application; YMMV.)

In one of the dynamoMessagingSystem.xml files in our project’s source code, as child elements of the patchbay XML element, there were message-source and message-sink elements (and sub-elements) that were (apparently) incorrectly defined, and also were not actually being used by the application. 

Removing those bad elements from the dynamoMessagingSystem.xml file, and then rebuilding and redeploying the application, resolved the issue.

Thanks to Miles from Oracle Support for his assistance in troubleshooting this issue!

Fix: Endeca Baseline fails with NullPointerException in pushAuthoringContentToDgraphById

Symptom 1: When running a Baseline Index in Endeca 3.1.1, the MDEX server(s) do appear to be updated, but several errors appear in the console output, the first of which is:

[05.29.13 14:06:10] INFO: Pushing authoring content to dgraph: AuthoringDgraph
[05.29.13 14:06:10] SEVERE: null
Occurred while executing line 5 of valid BeanShell script:
[[

2|
3| compId = invokingObject.getElementId();
4| log.info("Publishing Workbench 'authoring' configuration to MDEX '" + compId + "'");
5| IFCR.pushAuthoringContentToDgraphById(compId);
6|
7|
]]

[05.29.13 14:06:10] WARNING: Caught an exception while executing post-startup script for server component 'AuthoringDgraph'.

Caused by com.endeca.soleng.eac.toolkit.exception.AppControlException
com.endeca.soleng.eac.toolkit.script.Script runBeanShellScript - Error executing valid BeanShell script.
Caused by java.lang.NullPointerException
com.endeca.soleng.eac.toolkit.utility.IFCRUtility extractIfcrErrorMessage – null

Symptom 2: Running the control/promote_content.sh script fails with a similar error.

Symptom 3: Trying to add a new content item in the Endeca Workbench Rule Manager fails with the error “Last attempt to publish to the MDEX Engine failed”.  If the mouse cursor is hovered over the error message, a tooltip appears with an additional error message: “com.endeca.ifcr.publish.impl.PublishException: Failed to publish to the following dgraphs: [dgraph url]”.

Solution: This started happening for my new Endeca application after I changed the password for the Endeca Workbench “admin” user to something other than the default value of “admin”. 

Setting the correct (updated) password for the admin user in the application’s config/script/WorkbenchConfig.xml file resolved the issue.

Credit to Dan O. from Thanx Media for getting this issue tracked down.  Thanks Dan!

Oracle ATG – Endeca integration: Baseline Index process troubleshooting

This article contains troubleshooting steps for when an Endeca Baseline Index, as manually launched via the ATG Dyn Admin | ProductCatalogSimpleIndexingAdmin component, doesn't complete successfully.

All of this information is from my own experience and troubleshooting in trying to get an ATG-Endeca integration for a new ATG/Endeca application wired up and working successfully. (Which I did eventually accomplish, but not until after working through a fair number of unexpected issues, as should be apparent from the list below!) 

So, disclaimer: The suggested resolutions may or may not be “best practice” or otherwise work well for your own application. I’m just posting this information in hopes that it may be helpful; I certainly wished at points that there was more ATG-Endeca integration help available on the web at times, while I myself was initially troubleshooting my way through each one of these issues!

---

Q: The ProductCatalogSimpleIndexingAdmin doesn't exist in Dyn Admin. Searching for it returns 0 results. Browsing for it shows that the /atg/endeca subtree doesn't exist.

1. The "ATG-Endeca integration" feature might not be installed in ATG. This can be done with the atgdir/home/bin/sim.sh utility.

2. The application startup might not have included the required Nucleus modules for the Endeca indexing integration. In my situation, I found that I needed to add the following Make sure the following Nucleus components are included with the application server startup – these may or may not be the specific set of modules that are appropriate for your ATG application:

• DAF.Endeca.Index
• DCS.Endeca.Index

---

Q: In Dyn Admin, the ProductCatalogSimpleIndexingAdmin component does come up, but the special UI that normally appears at the top of that component's Dyn Admin page, that allows a Baseline Index to be launched and shows the status of the process, doesn't appear at all.

Try inspecting the application startup log for errors and warnings.  I saw this happen when my product-sku-output-config.xml file contained an extra “-->” token; this was reported as a parse error in the application startup logs.

---

Q: Errors about missing tables appear in the log, including table srch_site_content, both at server startup and when trying to run Baseline Index.

I ended up needing to manually run some ddl files in my local dev environment to resolve this. I ran:

On core:

• search_site_ddl.sql

On cata and catb:

• versioned_search_site_ddl.sql

These scripts are packaged with ATG. Look for them in one of the descendant directories of the ATG_HOME directory.

---

Q: Under JBoss, trying to run Baseline Index via ProductCatalogSimpleIndexingAdmin fails. Exception appears in the log: "atg.repository.search.indexing.IndexingException: java.lang.NoSuchMethodError: org.apache.cxf.transport.AbstractTransportFactory.<init>(Ljava/util/List;)"

This is apparently due to a version conflict between the version of CXF that comes packaged with JBoss, and the version of CXF that ATG assumes will be available to it.

I spent some time looking into how to disable or otherwise not use the CXF version bundled with JBoss (and use ATG's version of CXF instead), but never found a solution. I ended up switching my local environment to use WebLogic instead or JBoss as a workaround.

Here's a post on the Oracle forums with this same problem (and no solution, as of 5/2013): https://forums.oracle.com/forums/thread.jspa?messageID=10622176

---

Q: The second portion of the Baseline Index process (RepositoryExport) fails immediately.

Possible additional symptom in log: /atg/commerce/endeca/index/ProductCatalogSimpleIndexingAdmin ---java.lang.RuntimeException: org.apache.commons.httpclient.ConnectTimeoutException: The host did not accept the connection within timeout of 30000 ms

There might be a problem with the connectivity between ATG and Endeca CAS – possibly a configuration issue or networking issue. Check:

1. In the DimensionDocumentSubmitter, SchemaDocumentSubmitter, and DataDocumentSubmitter components, are the CASHostName and CASPort properties set properly? If not, make the changes in the corresponding .properties files for each of those components (in ATG 10.1.1), or in the /atg/endeca/index/IndexingApplicationConfiguration/ component (in ATG 10.1.2).

2. Try pinging the target Endeca CAS server from the command line on the ATG machine to verify that it is accessible on the network. 

3. Verify that DNS resolution isn’t misconfigured such that the CAS hostname is resolving to an IP address of some server other than the actual CAS server.

---

Q: The second portion of the Baseline Index process (RepositoryExport) fails immediately. The log includes "Response-Code: 500"

The "500" here appears to be an HTTP 500 ("Internal server error") being returned from Endeca CAS from the web service call. I've seen this happen several times when attempting to run a Baseline Update after a previous Baseline Update failed midway through, and when the indexing process is being run from the ATG CA server.

The only workaround I know of at this point is to bounce the CAS server. Quick instructions to bounce CAS (in my Endeca environment; specifics for your environment may vary):

1. ssh to the Endeca indexing server.
2. Change directory to endeca/CAS/version number/bin
3. ./cas-service-shutdown.sh
4. nohup ./cas-service.sh &

---

Q: The second portion of the Baseline Index process (RepositoryExport) hangs for a while, and then fails with no records successfully processed. An error related to network connectivity appears in the ATG server log.

1. Is CAS running? Log on to the Endeca indexing server and run:

ps -ef | grep cas

If there's no result, then start CAS.

2. Alternatively, this might indicate a DNS issue. Make sure that when you ping the Endeca indexing server from your ATG server, the correct IP address is being returned.

---

Q: java.io.IOException: http://[endeca host]/[endeca app name]en_en_dimvals/?wsdl returned response code 404

This appears to be caused by an incorrect defaultRecordStoreName and/or endecaBaseApplicationName entry in the DataDocumentSubmitter, DimensionDocumentSubmitter, and/or SchemaDocumentSubmitter components (ATG 10.1.1) or in the IndexingApplicationConfiguration component (ATG 10.1.2). Check on those values for those components in Dyn Admin and/or in the application’s .properties files.

One specific thing to check: If the language code (e.g. “en”) is being specified in the endecaBaseApplicationName property, this might result in a defaultRecordStoreName value with too many language code values, e.g. “MyAppenen_en_dimvals”, which isn't correct – too many "en".

---

Q: Exception in thread "index-/atg/endeca/index/commerce/ProductCatalogSimpleIndexingAdmin" java.lang.NullPointerException: Property value cannot be null (dimval.display_name)

This apparently indicates that one or more of the item-descriptor tags in customCatalog.xml (in the ATG project) are missing display-name attributes. (Note that some item-descriptor tags – particularly, the OOTB ones – inherit a display-name via the xml-combine="append" attribute value.)

Another symptom of this: In Dyn Admin, RepositoryTypeDimensionExporter component, under "show XML", one of the dimval.display_name entries in the XML shown has an empty PVAL.

---

Q: The ProductCatalogOutputConfig step reports success, but shows 0 records processed. The server log doesn't show any warnings.

Additional symptom: The Dyn Admin | Commerce Admin | Catalog Verification reports thousands of warnings like: "/atg/commerce/catalog/custom/CatalogVerificationService Product 434318 is not in any catalog. Cannot verify info objects."

Additional symptom: When looking the properties for any particular product via the ProductCatalog component in dyn admin, several properties are missing: ancestorCategories, ancestorCategoryIds, siteIds, parentCategoriesForCatalog, catalogs, computedCatalogs.

To fix:

• In Dyn Admin | Commerce Admin, run Catalog Update and then Basic Maintenance. Then, run Catalog Verification; it should report 0 errors, 0 warnings.
• If an error occurs during Basic Maintenance, it might be caused by an database deadlock. (You can look at the application log file to verify whether or not this is the case.) 

---

Q: ProductCatalogOutputConfig step fails after around 200 records processed. Application log shows: SQLException: Internal error: Cannot obtain XAConnection weblogic.common.resourcepool.ResourceLimitException: No resources currently available in pool

This apparently indicated that ATG was trying to use more connections than were available in WebLogic. I’m not sure what caused this when I saw it.  I ended up just retrying the baseline index (without changing any settings or bouncing the server), and it worked the second time.

---

Q: java.lang.OutOfMemoryError occurs after processing some, but not all, of the product records.

I worked around this by increasing the Java max heap size (for the ATG application) to 3G, and the MaxPermSize to 384m.

---

Q: Silent failure on the EndecaIndexing step (the last step). Nothing appears in the ATG application log after the RepositoryExport steps completed. The Dyn Admin GUI just reports "COMPLETE (Failed)".

1. This may indicate that the forge process failed on the Endeca side.  Try looking at the logs on the Endeca indexing server for details about the failure.

2. Verify that EAC is running on the Endeca indexing server.

3. Verify that in the EndecaScriptService (ATG 10.1.1) or ApplicationConfiguration (ATG 10.1.2) component, the endecaBaseApplicationName (10.1.1) or baseApplicationName (10.1.2) property has the correct value.

---

Q: Second portion of baseline index (RepositoryExport) fails midway through. Error in log: "/atg/search/repository/BulkLoader — atg.repository.search.indexing.IndexingException: com.endeca.itl.recordstore.ConcurrentWriteException: Write in progress with generation 12"

I found that a workaround for this was to restart CAS on the Endeca indexing server, then retry the baseline import from the ATG side.

Update 8/19/2013: Apparently this issue can be caused by an update lock not being released when a previous baseline index operation was manually canceled. See article "Cancelling the Baseline Indexing Job from SimpleIndexingAdmin does not Release Update Lock (Doc ID 1576472.1)" on the Oracle Support site for more information.

ATG 10.1.2–“FileNotFound” error using Endeca search from an InvokeAssembler droplet

Issue

In Oracle ATG Commerce 10.1.2, a JSP page in my project failing to properly retrieve results from Endeca Guided Search.  (The application is not using Endeca Experience Manager).  The following error appeared in the ATG application log file:

**** Error      Mon Apr 15 09:55:55 EDT 2013    1366034155171   /atg/endeca/assembler/droplet/InvokeAssembler   A problem occurred assembling the content for content item /services/guidedsearch. The response received was {contentUri=/services/guidedsearch, @type=ContentInclude, @error=FileNotFound}. Servicing the error open parameter.

Resolution

The problem turned out to be not a missing file, but that some of the Endeca-related ATG components were misconfigured.  We resolved the problem by making sure that the following ATG component properties had correct values:

atg.endeca.ApplicationConfiguration

• baseApplicationName
• defaultApplicationName
• keyToApplicationName
• defaultLanguageForApplications
• workbenchHostName
• workbenchPort

atg.endeca.assembler.AssemblerApplicationConfiguration

• defaultMdexHostName
• defaultMdexPort

atg.endeca.assembler.cartridge.manager.DefaultWorkbenchContentSource

• serverPort

atg.endeca.assembler.cartridge.manager.AssemblerSettings

• experienceManager

Update August 7, 2013

My team today encountered this "FileNotFound" error again when setting up a new environment. Even after verifying the ATG configuration was 100% correct, the problem persisted. The Endeca JSP Reference Application ("orange application") also returned search results successfully, which demonstrated that the Endeca MDEX server cluster was working ok.

The fix turned out being that the promote_content.sh script located at endeca/apps/appname/control needed to be run. After running that script and having it report successful completion, ATG immediately starting being able to retrieve search results from Endeca successfully (no service restarts needed).

Issue: Non-ASCII characters change to inverted question mark (“¿”) after save

I was recently entering some content into a Confluence wiki page that included some non-ASCII characters, such as “→” (U+2192, "RIGHTWARDS ARROW") and “←” (U+2190, LEFTWARDS ARROW).  (I can easily type those characters using Alt+Numpad 26 and Alt+Numpad 27 respectively, since my Windows XP machine is set to Code page 437.) However, after saving the document and coming back and looking at it later, all such characters had been changed to inverted (upside down) question mark characters, “¿”.

Investigation revealed that the problem was that the underlying Oracle database to which the content was being saved was set to use the Western European character set ISO-8859-1, as opposed to a more comprehensive character encoding such as UTF-8.  Since ISO-8859-1 doesn’t include the “leftwards arrow” or “rightwards arrow” characters, Oracle converted those characters to the inverted question mark instead.

I wanted to share this here since this information might be helpful to anyone encountering a similar issue in a variety of different possible applications, not necessarily just Confluence.