Geoportal Helpdesk - Support #2339 · i.A. Bernhard Jackl _____ Bundesamt für Kartographie und Geodäsie Referat GI8 - Geodateninfrastrukturleistungen Richard-Strauss-Allee 11, 60598

Geoportal Helpdesk - Support #2339DE - BKG: Germany Discovery Service not responding27 Feb 2015 07:33 pm - Angelo Quaglia

Status: Closed Start date: 27 Feb 2015Priority: Normal Due date:Assignee: Angelo Quaglia % Done: 0%Category: Harvesting process Estimated time: 0.00 hourTarget version:Submitting Organisation:DE - BKG Knowledge-Base relevant?:Proactive: Yes Keyword #1:Country: DE - Germany Keyword #2:Originating UI: Keyword #3:Description

From: Angelo Quaglia [mailto:[email protected]]Sent: 27 February 2015 07:48To: '[email protected]'; [email protected]; '[email protected]'Cc: '[email protected]'; '[email protected]'; '[email protected]';'[email protected]'; '[email protected]'Subject: RE: problematic harvesting last night by the INSPIRE Geoportal

Dear Mr. Jackl,

The harvesting has been suspended until we receive some feedback from you (no reply has been received to our last email)

Yesterday’s harvesting went even worse than the previous one ( which we wrote about in our last email http://inspire-geoportal.ec.europa.eu/resources/errors/INSPIRE-e70f8705-e89d-11e2-9634-52540004b857_20150219-210312/servics/1/PullResults/

):

http://inspire-geoportal.ec.europa.eu/resources/errors/INSPIRE-e70f8705-e89d-11e2-9634-52540004b857_20150226-210554/servics/1/PullResults/

/system/rich/rich_files/rich_files/000/000/056/original/mail4.png

21 Mar 2019 1/25

http://inspire-geoportal.ec.europa.eu/resources/errors/INSPIRE-e70f8705-e89d-11e2-9634-52540004b857_20150219-210312/services/1/PullResults/




The status has been reverted to the last reasonably good harvesting:

http://inspire-geoportal.ec.europa.eu/resources/INSPIRE-e70f8705-e89d-11e2-9634-52540004b857_20150206-223829/services/1/PllResults/


As said above, the harvesting has been suspended until we receive some feedback from you.

Best regards,

Angelo Quaglia

Ing. Angelo Quaglia

External Consultant

European Commission, DG Joint Research CentreInstitute for Environment and Sustainability

Digital Earth and Reference Data Unit, T.P. 262

Via E. Fermi, 2749.I-21027 Ispra (VA)Italy

Tel: +39 347 78 88 492Fax: +39 0332 78 6325e-mail: mailto:[email protected]

URL: http://ies.jrc.ec.europa.eu/SDI/sdi-about-us/staff-profiles/angelo-quaglia.html

21 Mar 2019 2/25

http://inspire-geoportal.ec.europa.eu/resources/INSPIRE-e70f8705-e89d-11e2-9634-52540004b857_20150206-223829/services/1/PullResults/


mailto:[email protected]

http://ies.jrc.ec.europa.eu/SDI/sdi-about-us/staff-profiles/angelo-quaglia.html

The views expressed are purely those of the writer and may not in any circumstances be regarded as stating an official position of theEuropean Commission.

History#1 - 27 Feb 2015 07:35 pm - Angelo Quaglia

From: Support Geoportal [mailto:[email protected]]Sent: 27 February 2015 12:47To: [email protected]: Re: [Ticket#2015022710000033] problematic harvesting l [...]

Dear Mr. Quaglia,

due to a Denail of Service (DoS) attack, the service have been pretty slow, yesterday. I apologise for the dramatic slowdown, the responsibleIT-Department is currently investigating the problem. I cannot go further than that at this stage but I will keep inform you when I have more information.

Apart from this fact, the harvesting between Germany's National Discovery Service and INSPIRE Discovery Service is indeed gradually slowed down.The response time is according to our monitoring, directly at the csw-server, obviously within 3 seconds.It's difficult to clarify the reason of additional time causing the extended response time.

However the IT-Department is being involved to guarantee a stable service within a reasonable amount of time.

Did you observed similar slowdown effects with other national catalogues?

Kind regards,Bernhard Jackl

Mit freundlichen Grüßen

Im AuftragBernhard Jackl

________________________________________Support Geoportal.deBundesamt für Kartographie und Geodäsie (BKG) Richard-Strauss-Allee 11D-60598 Frankfurt am MainE-Mail: [email protected]: www.geoportal.de

#2 - 27 Feb 2015 07:37 pm - Angelo Quaglia

21 Mar 2019 3/25


http://www.geoportal.de

From: Angelo Quaglia [mailto:[email protected]]Sent: 27 February 2015 13:11To: Support GeoportalSubject: Re: [Ticket#2015022710000033] problematic harvesting l [...]

Dear Mr. Jackl,

All other services have been harvested regularly yesterday with the exception of the Check republic because their system suffered from a fatalsoftware error.

Just tell us when we can reenable the harvesting.

It might be worth setting up a monitoring service on an external service.

Best regards,

Angelo Quaglia

Sent from my iPhone

#3 - 27 Feb 2015 07:38 pm - Angelo Quaglia

From: [email protected] [mailto:[email protected]]Sent: 27 February 2015 13:24To: [email protected]: [email protected]; [email protected]; [email protected]; [email protected];[email protected]; [email protected]: AW: problematic harvesting last night by the INSPIRE Geoportal

Dear Mr. Quaglia,

you can find further details in Ticket 2015022710000033.

Best regards,

Bernhard Jackl


21 Mar 2019 4/25

i.A. Bernhard Jackl________________________Bundesamt für Kartographie und GeodäsieReferat GI8 - Geodateninfrastrukturleistungen

Richard-Strauss-Allee 11, 60598 Frankfurt am MainTelefon: +49 (0)69 6333-338

#4 - 27 Feb 2015 07:44 pm - Angelo Quaglia- Subject changed from Germany Discovery Service not reponding to Germany Discovery Service not responding

#5 - 03 Mar 2015 05:46 pm - Angelo Quaglia

From: Support Geoportal [mailto:[email protected]]Sent: 03 March 2015 17:39To: [email protected]: Re: [Ticket#2015022710000033] problematic harvesting l [...]

Dear Mr. Quaglia,

the responsible it-department fixed the Denail of Service attack, so I kindly ask you to reenable the harvesting.

Many thanks in advance!

Best regards,Bernhard Jackl


Im AuftragBernhard Jackl

________________________________________Support Geoportal.deBundesamt für Kartographie und Geodäsie (BKG) Richard-Strauss-Allee 11D-60598 Frankfurt am MainE-Mail: [email protected]: www.geoportal.de

#6 - 03 Mar 2015 05:59 pm - Angelo Quaglia- Status changed from Assigned to Resolved

21 Mar 2019 5/25



From: Angelo Quaglia [mailto:[email protected]]Sent: 03 March 2015 17:59To: 'Support Geoportal'Cc: [email protected]; 'Michael Lutz' ([email protected]); 'Freddy Fierens' ([email protected])Subject: RE: [Ticket#2015022710000033] problematic harvesting l [...]

Dear Mr. Jackl,

The harvesting for Germany has been re-enabled and it will run, as previously agreed, during the first available slot on Thursday or Friday after 9pm.

Many thanks and best regards,

Angelo Quaglia

#7 - 09 Mar 2015 08:15 am - Angelo Quaglia- Status changed from Resolved to Assigned

From: Angelo Quaglia [mailto:[email protected]]Sent: 09 March 2015 08:15To: '[email protected]'Cc: '[email protected]'; '[email protected]'; '[email protected]'; '[email protected]';'[email protected]'; '[email protected]'Subject: RE: problematic harvesting last night by the INSPIRE GeoportalImportance: High

Dear Mr. Jackl,

The harvesting is still problematic.

Could you please ask someone in your technical team to review the error messages?

21 Mar 2019 6/25



Best regards,

Angelo Quaglia

Ing. Angelo Quaglia

External Consultant






The views expressed are purely those of the writer and may not in any circumstances be regarded as stating an official position of the EuropeanCommission.

#8 - 03 Apr 2015 05:31 pm - Angelo Quaglia

From: Angelo Quaglia [mailto:[email protected]]Sent: 03 April 2015 17:30

21 Mar 2019 7/25




To: '[email protected]'Cc: '[email protected]'; '[email protected]'; '[email protected]'; '[email protected]'Subject: RE: AW: problematic harvesting last night by the INSPIRE Geoportal

Dear Mr. Jackl,

The latest harvesting ran yesterday but only half of the metadata documents could be retrieved:

Result of the interaction with the Discovery Service

Resources available for discovery: 27612, Expected Resource Count: 27612, Actual Resource Count : 13450



Best regards,

Angelo

Ing. Angelo Quaglia

External Consultant






21 Mar 2019 8/25





#9 - 08 May 2015 02:41 pm - Angelo Quaglia- Submitting Organisation set to Germany - BKG

From: [email protected] [mailto:[email protected]]Sent: 08 May 2015 14:02To: [email protected]: [email protected]; [email protected]; [email protected]; [email protected];[email protected]; [email protected]: WG: update catalogue-software on FR 08.05.15

Dear Mr. Quaglia,

our maintenance ended today at 11.00 (UTC+2) a.m.

Our catalogue-service is now again fully available.

The best way is to harvest following URL : http://gdk.gdi-de.org/inspire/srv/eng/csw?SERVICE=CSW&VERSION=2.0.2&REQUEST=GetCapabilities

Best regards,

Bernhard Jackl

___________________________________________

Referat GI8 - Geodateninfrastrukturleistungen

Bundesamt für Kartographie und Geodäsie (BKG)

Richard-Strauss-Allee 11, 60598 Frankfurt am Main

Telefon: +49 (0)69 6333-338

Fax: +49 (0)69 6333-441

E-Mail: [email protected]

Internet: http://www.bkg.bund.de

21 Mar 2019 9/25

http://gdk.gdi-de.org/inspire/srv/eng/csw?SERVICE=CSW&VERSION=2.0.2&REQUEST=GetCapabilities


http://www.bkg.bund.de/


#10 - 08 May 2015 02:43 pm - Angelo Quaglia

From: Angelo Quaglia [mailto:[email protected]]Sent: 08 May 2015 14:40To: '[email protected]'Cc: '[email protected]'; '[email protected]'; '[email protected]'; '[email protected]';'[email protected]'; '[email protected]'Subject: RE: update catalogue-software on FR 08.05.15

Dear Mr. Jackl,

You need to kindly update the registration at http://inspire-geoportal.ec.europa.eu/INSPIRERegistry/ if you would like the INSPIRE Geoportal to use thenew endpoint.

I have tested the new endpoint and found two issues:

1. The capabilities retrieved from http://gdk.gdi-de.org/inspire/srv/eng/csw?SERVICE=CSW&VERSION=2.0.2&REQUEST=GetCapabilities

contain the old endpoints, so the geoportal will be forced to use those:

http://ims.geoportal.de/gdi-de/srv/eng/csw"/>


And

21 Mar 2019 10/25

http://inspire-geoportal.ec.europa.eu/INSPIRERegistry/

http://gdk.gdi-de.org/inspire/srv/eng/csw?SERVICE=CSW&VERSION=2.0.2&REQUEST=GetCapabilities



1. If I request the first 50 records:

http://www.opengis.net/cat/csw/2.0.2" xmlns:ogc="http://www.opengis.net/ogc" xmlns:gmd="http://www.isotc211.org/2005/gmd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:apiso="http://www.opengis.net/cat/csw/apiso/1.0" service="CSW" version="2.0.2"maxRecords="50" startPosition="1" resultType="results" outputSchema="http://www.isotc211.org/2005/gmd" outputFormat="application/xml">

full

http://www.opengis.net/ogc">

AnyText

inspireidentifiziert

I only get back 49 records:

http://www.opengis.net/cat/csw/2.0.2" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="

21 Mar 2019 11/25

http://www.opengis.net/ogc

http://www.isotc211.org/2005/gmd

http://www.w3.org/2001/XMLSchema-instance

http://www.opengis.net/cat/csw/apiso/1.0



http://www.opengis.net/cat/csw/2.0.2 http://schemas.opengis.net/csw/2.0.2/CSW-discovery.xsd">

http://www.isotc211.org/2005/gmd" xmlns:gco="http://www.isotc211.org/2005/gco" xmlns:gml="http://www.opengis.net/gml" xmlns:xlink="http://www.w3.org/1999/xlink"


Best regards,

Angelo Quaglia

Ing. Angelo Quaglia

External Consultant







21 Mar 2019 12/25

http://www.opengis.net/cat/csw/2.0.2%20http:/schemas.opengis.net/csw/2.0.2/CSW-discovery.xsd

http://schemas.opengis.net/csw/2.0.2/CSW-discovery.xsd

http://www.isotc211.org/2005/gco

http://www.opengis.net/gml

http://www.w3.org/1999/xlink




From:[email protected] [mailto:[email protected]]Sent: 08 May 2015 16:28To:[email protected]:[email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]: AW: update catalogue-software on FR 08.05.15

Dear Mr. Quaglia,

thanks for the response.

The central-nodes are exactly the same. The URL of the xlink:hrefs are used as an alias.

Concerning the GetRecords Request, I can’t reproduce the response of the poster, as you will see from the attachment. I’ll kindly ask you to test it onceagain.

I’ve been registered as bjackl but I can’t make any changes, maybe I need further privileges or I may have missed some details.

/system/rich/rich_files/rich_files/000/000/188/original/mail129.png/system/rich/rich_files/rich_files/000/000/187/original/mail130.png

Best regards,

Bernhard Jackl

___________________________________________

Referat GI8 - Geodateninfrastrukturleistungen

Bundesamt für Kartographie und Geodäsie (BKG)


Telefon: +49 (0)69 6333-338

Fax: +49 (0)69 6333-441

E-Mail: [email protected]

Internet: http://www.bkg.bund.de


21 Mar 2019 13/25











http://www.bkg.bund.de/



Dear Mr. Jackl,

Therefore I assume that the INSPIRE Geoportal can safely keep using the previous endpoint.

I do not see the filter in your GetRecordsRequest:


This is the filter registered for Germany:


full


AnyText


21 Mar 2019 14/25






If you include that, you will reproduce the issue:


Often that means that one of the records has a critical issue and the service does not manage to emit it.

Usually, you should be able to spot the error in the log files of the server.

Best regards,

Angelo Quaglia



Regarding the access to the registry,

The three available user accounts are allocated as follows:


We need an email sent by INSPIRE National Contact Point for Germany asking for the appropriate replacements.

You can find detailed instructions here:

21 Mar 2019 15/25

https://ies-svn.jrc.ec.europa.eu/projects/mig-inspire/wiki/DiscoveryServiceRegister

Best regards,

Angelo Quaglia


From: [email protected] [mailto:[email protected]]Sent: 08 May 2015 16:38To: [email protected]: Automatische Antwort: update catalogue-software on FR 08.05.15

Vielen Dank für Ihre Nachricht! Ich bin voraussichtlich ab dem 01.06.2015 wieder erreichbar. Ihre E-Mail wird nicht weitergeleitet. In dringenden Fällenwenden Sie sich bitte an: [email protected].

Mit freundlichen Grüßen,

Bernhard Jackl

I'm out offiice until 01.06.2015. Your email will not be processed. If you have any urgent matter please contact [email protected]

Best regards,

Bernhard Jackl

#15 - 09 Jun 2015 01:15 pm - Angelo Quaglia

From:[email protected] [mailto:[email protected]]Sent: 03 June 2015 17:09To:[email protected]:[email protected]; [email protected]; [email protected]: Geodatenkatalog.de

Dear Mr. Quaglia,

21 Mar 2019 16/25

https://ies-svn.jrc.ec.europa.eu/projects/mig-inspire/wiki/DiscoveryServiceRegister









We had a conversation in Lisbon last week. We were talking about the performance of the German Catalogue-Service for INSPIRE. By the way, Thankyou for your time!

I explained that we (BKG/Germany) were successful with external testings of our catalogue-service. We had no problem with performance andtime-outs. On the other hand you showed me test results from JRC, which showed that you have some trouble by harvesting our catalogue.

I promised to ask my colleagues for providing some documentation about our tests. They are informed and will soon get in touch with you. I hope thatwill support your efforts for improving the harvesting process.

If you have any further question, don’t hesitate contacting us,

best regards

Martin Lenk

--

Koordinierungsstelle GDI-DE / SDI Coordination Office

Bundesamt für Kartographie und Geodäsie / Federal Agency for Cartography and Geodesy


Germany

Telefon: +49 (0)69 6333 300

Fax: +49 (0)69 6333 446

E-Mail (request): [email protected]

E-Mail (information): [email protected]

Internet: www.gdi-de.org | www.geoportal.de


Von: Angelo Quaglia [mailto:[email protected]]

21 Mar 2019 17/25

http://dict.leo.org/ende/index_en.html#/search=colleague&searchLoc=0&resultOrder=basic&multiwordShowSingle=on



http://www.gdi-de.org



Gesendet: Mittwoch, 3. Juni 2015 18:17An: Lenk, MartinCc: Heimbürger, Olaf; Jackl, Bernhard; KstGDIDEBetreff: RE: Geodatenkatalog.de

Dear Mr.Lenk,

Many thanks for coming back to me.

Here are some details:

In general, the service responds very well, with initial responses as low as 1.5 seconds throughout the whole operation.

Sporadically, the initial response time peaks over 21 seconds and it is difficult to explain why.

The last harvesting operation performed on 28th May, saw 6 batches timing-out after 21 seconds (time is shown in UTC):

2201-2250 (/system/rich/rich_files/rich_files/000/000/210/original/mail139.png28 May 2015, 19:47:55 /system/rich/rich_files/rich_files/000/000/209/original/mail140.png28 May 2015, 19:49:05)

2251-2300 (/system/rich/rich_files/rich_files/000/000/210/original/mail139.png28 May 2015, 19:49:07 28 May 2015, 19:50:17)





/system/rich/rich_files/rich_files/000/000/210/original/mail139.png = start

/system/rich/rich_files/rich_files/000/000/209/original/mail140.png = end

Of course, something could have happened on our side or along the internet connection.

However, it is not easy to explain why a similar issue happened in two different timeframes.

21 Mar 2019 18/25

We can increase the initial timeout which is now set to 15 seconds.

When a request fails it is retried up to four times incrementing the timeout.

Some inputs from your colleagues will be much appreciated.

Best regards,

Angelo Quaglia


From:[email protected] [mailto:[email protected]]Sent: 08 June 2015 11:25To:[email protected]:[email protected]; [email protected]: AW: Geodatenkatalog.de

Hallo Mr. Quaglia,

can you send us your csw-request (post or get). I will try to reproduce this behavior

with our test-tool inside and outside of our it-infrastructure.

Best regards

Dr. Olaf Heimbürger

------------------------------------------------------------------------------------------

Fedral Agency of Cartography and Geodesy

Division Geoinformation

21 Mar 2019 19/25






Geospatial Infrastructure Services

Operation Center GDI-DE

Richard-Strauss-Allee 1160598 Frankfurt/Main------------------------------------------------------------------------------------------Internet: http://www.bkg.bund.de@mail: [email protected].: +49 69 6333 319Fax: +49 69 6333 441


From: Angelo Quaglia [mailto:[email protected]]Sent: 09 June 2015 13:14To: '[email protected]'Cc: '[email protected]'; '[email protected]'Subject: RE: Geodatenkatalog.de

Dear Mr. Heimbürger,

the Geoportal always uses HTTP POST requests like the following one:


full


AnyText


21 Mar 2019 20/25

http://www.bkg.bund.de







5 concurrent requests are fired until all the catalogue has been harvested.

The slowdowns happen at different times and for different baatches.


19801-19850 /system/rich/rich_files/rich_files/000/000/210/original/mail139.png05 Jun 2015, 02:07:56 /system/rich/rich_files/rich_files/000/000/209/original/mail140.png05 Jun 2015, 02:09:09

19851-19900 05 Jun 2015, 02:08:09 05 Jun 2015, 02:09:22

19901-19950 05 Jun 2015, 02:08:25 05 Jun 2015, 02:09:39

19951-20000 05 Jun 2015, 02:08:35 05 Jun 2015, 02:09:47

20001-20050 05 Jun 2015, 02:08:49 05 Jun 2015, 02:10:01

20051-20100 05 Jun 2015, 02:09:10 05 Jun 2015, 02:10:22

20101-20150 05 Jun 2015, 02:09:27 05 Jun 2015, 02:10:39

20151-20200 05 Jun 2015, 02:09:41 05 Jun 2015, 02:10:52

20201-20250 05 Jun 2015, 02:09:49 05 Jun 2015, 02:10:59

20251-20300 05 Jun 2015, 02:10:02 05 Jun 2015, 02:11:13

20301-20350 05 Jun 2015, 02:10:26 05 Jun 2015, 02:11:4520351-20400 05 Jun 2015, 02:10:40 05 Jun 2015, 02:11:53

20501-20550 05 Jun 2015, 02:11:15 05 Jun 2015, 02:12:26

20401-20450 05 Jun 2015, 02:10:54 05 Jun 2015, 02:12:08

20451-20500 05 Jun 2015, 02:11:01 05 Jun 2015, 02:12:17

20501-20550 05 Jun 2015, 02:11:15 05 Jun 2015, 02:12:26

21 Mar 2019 21/25


After that the initial response time decreased gradually (9, 8, 7, etc.) until it went back to normal.

During the previous harvesting, the slowness occurred at different times for different batches:

2201-2250 (28 May 2015, 19:47:55 28 May 2015, 19:49:05)

2251-2300 (28 May 2015, 19:49:07 28 May 2015, 19:50:17)

2301-2350 (28 May 2015, 19:50:19 28 May 2015, 19:51:29)

2351-2400 (28 May 2015, 19:50:20 28 May 2015, 19:51:30)

6151-6200 (28 May 2015, 23:17:42 28 May 2015, 23:18:54)

6201-6250 (28 May 2015, 23:17:42 28 May 2015, 23:18:54)

As a side note, the current service capabilities (http://gdk.gdi-de.org/gdi-de/srv/eng/csw?service=CSW&request=GetCapabilities&version=2.0.2)

contain invalid content:

…

$INSPIRE_LOCALE

21 Mar 2019 22/25

http://gdk.gdi-de.org/gdi-de/srv/eng/csw?service=CSW&request=GetCapabilities&version=2.0.2

Best regards,

Angelo Quaglia


From: [email protected] [mailto:[email protected]]Sent: 09 June 2015 15:19To: [email protected]: [email protected]; [email protected]: AW: Geodatenkatalog.de

Thank you.

#20 - 06 Jul 2015 08:34 am - Angelo Quaglia- Status changed from Assigned to Resolved

From: Angelo Quaglia [mailto:[email protected]]Sent: 06 July 2015 08:33To: '[email protected]'Cc: '[email protected]'; '[email protected]'Subject: RE: todays harvesting CSW German SDI

Dear Mr. Jackl,

The latest harvesting went very well as all the records could be fetched.

The initial response time was excellent: less than two seconds, well below the three seconds limit, throughout all the harvesting process and onlyoccasionally peaked to 9.357.

21 Mar 2019 23/25



Many thanks for this significant improvement and your cooperation.

Best regards,

Angelo Quaglia

Ing. Angelo Quaglia

External Consultant







#21 - 15 Jul 2015 08:09 am - Angelo Quaglia- Status changed from Resolved to Closed

#22 - 23 Aug 2016 06:07 pm - Angelo Quaglia- Subject changed from Germany Discovery Service not responding to DE - BKG: Germany Discovery Service not responding- Submitting Organisation changed from Germany - BKG to DE - BKG

#23 - 24 Aug 2016 05:48 pm - Angelo Quaglia- Category set to Discovery Service Register

21 Mar 2019 24/25




#24 - 31 Aug 2016 09:03 am - Angelo Quaglia- Category changed from Discovery Service Register to Harvesting process

#25 - 12 Sep 2016 11:19 am - Angelo Quaglia- Proactive set to Yes

#26 - 21 Sep 2016 06:31 pm - Angelo Quaglia- Country set to DE - Germany

21 Mar 2019 25/25

Documents

Geoportal Helpdesk - Support #2339 · i.A. Bernhard Jackl _____ Bundesamt für Kartographie und Geodäsie Referat GI8 - Geodateninfrastrukturleistungen Richard-Strauss-Allee 11, 60598