The other day one of our Plan Academy members called our support line with an unusual P6 problem. I’m always happy to help out our members solve P6 problems, even when it’s 4:30pm on the Friday afternoon before I’m off on a full week of vacation….
OK. Full disclosure here.
I had already powered down my laptop for the weekend and was just about to grab something ice-cold and carbonated when the phone rang. I was hoping it was something we could solve quick or that I had already written a solution for. I was tired, and it was hot in the office. The patio was calling. This wasn’t how I was expecting to wrap up my week. Oh well.
The call was was from Doug (not his real name) from Alberta, a Plan Academy member who works for a large national construction company…and he was having problems with an XER file he was sending to a client. Doug’s client was trying to import Doug’s XER File, but it was taking hours to import Not an exaggeration – Doug said it took 6 hours for the client to import his P6 XER file successfully. Something was clearly wrong.
Sometimes you get those problems that just can’t stay away from because it’s something you haven’t seen before and curiousity just takes over. That’s what happened here for me. I was thinking “6 hours to import? How can that possibly be?”
So I made a bargain with myself. If I could help Doug in the next 30 minutes (by 5pm), then I would reward myself with not one, but TWO ice-cold carbonated bevvies. I would be on vacation after all and sometimes you have to motivate yourself!
Now Doug described his project schedule to me. It was a few thousand lines – not uncommon – and it was both resource and cost-loaded. He and his client were using Primavera P6 Professional v8.4.
I did some quick research online about slow importing of XER Files, but there wasn’t any obvious leads there. Here’s what I suggested to Doug.
XER File Slow To Import Into Primavera P6? POBS XER File Data May Be the Culprit
Doug’s XER File was clearly taking too long to import. In this case, I can see 2 things that might need addressing:
- The XER File is is having issues.
- The client’s machine or database is having issues.
My experience is that most machines and databases don’t run into slow performance issues like they used to 10+ years ago…back in the day when you had too many background processes running on your PC and those programs would grind the processor to a halt. So I was thinking there was likely something going on with the XER file.
Here’s what I recommended to Doug:
A) Rather than use the XER File Format, try the XML format that P6 supports instead.
The XML format functions much like the XER File format. There are some differences, but the project will come across regardless. If the XML file imports quickly, then there’s a problem with the XER file.
B) Remove the POBS Data from the XER File.
This solution was suggested recently on PlanningPlanet where this POBS issue came to my attention (and thanks to the smart P6 experts who suggested it there). I’ll outline the steps to easily removing POBS data soon, including a tip that will be a major timesaver, but first, what did Doug do?
Doug opted to try suggestion B). By removing the POBS data from his XER File, we found that he was able to reduce his file’s size from 25Mb down to 2Mb! Wow! That’s a lot of POBS data!
For Doug, removing the P6 POBS data also reduced the import time from 6 hours to >2 minutes! That’s an exponential improvement!
Yes, we got Doug fixed and I made it to the patio and kicked off a great vacation week. But I’m not finished with this POBS issue. I’m not completely sure why it’s cropping up right now, but I’ve seen enough online grumblings about it to want to tackle it in detail.
So let’s dig in and get down to the bottom of POBS data and XER files in Primavera P6.
What is POBS Data?
Since this is a fairly new issue with P6, I’ve done some homework around the POBS data issue.
Every Primavera P6 database has a POBS table. Looking at the Primavera P6 database schema docs, POBS stands for Performing Organizational Breakdown Structure. The POBS data isn’t used (yet) by Primavera P6 Professional or Primavera P6 EPPM. Here’s what an Oracle Support doc titled “What is the POBS Table Used for and What Data is Stored in POBS? (Doc ID 1959228.1)” had to say:
Functions related to table POBS have not been implemented yet so the table has not been put to use. The table may be removed in a future release.
If there is data in this table, then it’s probably present in the sample database and not inserted during operations performed in the application.
That doc says it applies to Primavera P6 EPPM v7.0.3.0 or later.
What are the problems with POBS data & XER Files?
Another support doc discussed how the POBS table is getting overly large and affecting performance of the application. Let’s cover that issue another time.
However, at least 1 Bug has been logged for performance issues related to POBS data (Bug 20800033 : EXCLUDE POBS DATA FROM XER EXPORT & IMPORT OPERATIONS), and the Bug had this to say about XER import issues:
We do not utilize the POBS table yet we export/import the data from this
table when completing XER Export/Import. The XER export/import should be
written to exclude this data with XER export/import operations of P6
Professional.
So essentially, here’s what we know:
- POBS data exists in the POBS table in the P6 database
- POBS data is exported and imported via XER file
- POBS data isn’t used for anything
- Too much POBS data can affect performance of import (and maybe export)
I have a hunch that POBS data is related to Primavera Cost Manager, a now-defunct cost management solution that was once part of the Primavera suite years ago. Primavera Cost Manager integrated tightly with P6 Professional and looking at documentation that still exists, there was a concept of a Performing Organizational Breakdown Structure in Cost Manager. But alas, only Oracle can confirm this.
How Can I Clean POBS Data From My XER File?
[NOTE: a simple alternate solution is provided at the bottom of this post]
We’ve already shown you how can you clean global data from XER Files with the XER File Parser, but this solution is much simpler. Let me start by saying you can do this the s..l..o…w way or the quick way (my way). Let me show you how it’s done.
First, export your project to an XER file.
Since XER files are actually text files, you can edit them with a text editor like notepad. Here’s what one looks like when you open it up:
We’re going to delete a section of data from the file. To denote a new section, look for a “%T”. To find the POBS section, look for “%T POBS”. That indicates the start of the POBS data in your file. You should see it somewhere near the top of the file in first 100 lines. Above, you can see it on line 29 of the file.
Deleting POBS Data
Basically, the process is to delete all lines starting from (and including) the “%T POBS” line, until the next “%T” in the file. Do not delete the next “%T” line.
So if you’re using notepad, you can simply highlight this huge swath of text and hit delete.
In my example above, there’s less than 200 lines to highlight so it’s pretty easy.
But you’ll see that the amount of POBS data isn’t consistent. Here’s a short 30-activity project schedule as viewed in a text viewer.
The image above might be more typical. The POBS data spans from line 29 to line 4412. Most of this file is POBS data!
So if you’ve got a huge file and a huge amount of POBS to clear out of your XER file, let me know you the quick way to clean it.
The Quick Way to Clean POBS Data From a P6 XER File
Let’s begin. First, you’ll need some better text editing tools than simple Windows Notepad. I’m recommeding Notepad++; free and easy to use. It’s worth the download!
1) Make a backup of your XER File first. Edit your file in Notepad++.
2) Find the “%T POBS” line in your file – somewhere in the first 50 lines. Note what line number this is, you’ll need it later.
3) Now, let’s find the next “%T” line. Place your cursor at the end of the “%T POBS” line. Hit CTRL+F (or use the Search menu) and search for the next “%T” occurrence in the file.
4) Make a note of the line number you’ve been taken to. Mine was line 4415.
5) Now we’re going to use some tools to highlight the POBS data since we know the line numbers. Go back to the “%T POBS” line. Place your cursor at the beginning of the line. Now Goto Edit -> Begin/End Select. This essentially tells Notepad++ to start highlighting here.
6) Now hit CTRL+G (or goto Search -> Goto). Make sure to choose Line and type in the line number from step 4) – for me 4415.
7) Now that you’re on the next %T line, tell Notepad++ to end the highlighting here. Goto the Edit menu and uncheck Begin/End Highlighting.
8) Now all of your POBS data is highlighted. Simply tap Delete or Backspace to remove it from the file. Make sure you delete the %T POBS line as well, but not any other lines.
9) Save the file and import.
For the xer file I used in this writeup, removing the POBS data took it from 4489 lines down to 102 lines. And it reduced the file from 564KB down to a mere 21KB.
Why Are Primavera P6 Users Having Issues With POBS Now?
That’s a tough question.
Reports from Oracle say that once Primavera P6 POBS data is in your database, it will be exported along with an XER File. The receiving database will then “be infected” with this data. The data seems to build up in the database (something I’ll cover fixing in my next post) and continues to grow as files go back and forth.
But why is this suddenly an issue now?
One theory I have is that a recent release, possibly Primavera P6 v8.4, had POBS data in the sample database that is often setup at install time. I’m choosing v8.4 as it was referenced in the Bug I mentioned earlier in this post. Users often use the sample database as a production database and the data could be propagating from there.
Whatever my theories are, these POBS issues seem to be appearing more and more.
Conclusion
If you’re having difficulties with performance importing or exporting Primavera P6 XER files, try cleaning the POBS data from them. Yes, you can use windows Notepad, but if the file is large, you’ll be scrolling for an hour. Try using the techniques I’ve shown above along with the Notepad++ text editor to save you time.
I fully recognize that what I’m offering here isn’t truly a permanent solution. Constantly editing XER files to have a reasonable import time isn’t viable and it is a product issue that needs to be addressed by Oracle. Here’s hoping this gets you by in the meantime.
In my next post, I discuss the POBS table in your Primavera P6 database and strategies to keep it clean or empty. Read it now – How to Clean Your P6 POBS Table On MSSQL & Oracle DB.
Update – 2016
It turns out that using notepad is really too tedious, although some commenters on this post had some great suggestions for making that process easier. But let me make life even easier by recommending a simple utility created by Plan Academy reader and senior planning engineer Ahmet Kochan. Ahmet’s POBS Cleaner is dead simple to use and free! POBS Cleaner can clean 1 or more XER files at the same time. It creates a newly cleaned version of your XER file with the extension __cleaned.xer in case you ever need the original file.
There are some other tools on the market that clean POBS from XER files, but Ahmet’s tool is free and so simple to use that I highly recommend it.
Download POBS Cleaner
Enjoy! POBS Cleaner is even packaged here with a test project so you can see how it works before you use it on a real project.
Michael
I’m using POBS Cleaner from Emerald Associates. I was using your quick method, but this is even quicker. It’s a free download and scrubs a 25mb file clean in less time than it takes to open the .xer in Notepad. There is also a POBS cleaner utility in xertoolkit
Derek, thanks for the heads up on these tools.
Michael/Derek,
Thanks for the info. and the tips for the solution/work around.
Great article! Thanks a lot!
Let me just add a bit of automation for this removal using standard MS Word means:
1) Open XER file through Word Open dialog with wildcard “*.*”
2) Select “Plain text” for the option of opening dialog
3) Use Search & Replace Dialog with the following settings
Search for: %T^tPOBS*%T
Replace with: %T
Use wildcards: +
4) Press Replace
5) Save as plain text file with standard win encoding page
6) In Explorer – Rename extension from txt into .xer
7) Enjoy 🙂
Ilya, interesting approach. Tried this, but my version of Word doesn’t offer me a “plain text” option anywhere – so the xer file doesn’t open properly. Any ideas?
I was kind of curious on tour post; I am a regular reader of this posts, but I can share with all of you a courious experience from the past: there was 2003 or 2004 maybe, and I has exactly the very same situation, but there were not hours, the hours became a weekend trying to import a file into a powerful laptop. In a nutshell, the issue was not the number of lines of the schedule, but the type of resource availability assigned per activity (I mean, many resources per activity). What I discovered, per a desperate mistake or casualty was this: if I were able to export first of all from the source all the resources in an .XER file and then import them in the new database, it was more or less quick (by that time, it took one minute…) then I exported the project and imported it into the new database, and it took only five minutes the import who had lasted days!!!!!.
Well, this happy coincidences happens. Hope this can become a hint for future cases, in particular working with old Primavera Setups, migrations and that type of cases.
Regards everybody. Marcos.
Marcos, thanks for sharing your experience and solution with us. We’ll have to see if we can confirm that the issues you mention from 2003/04 have been fixed in newer versions of Primavera P6 Professional.
Thanks again!
Michael
I created a tool to clean POBS data from XER files. Multiple XER files can be selected and cleaned at once. Original files will be kept and new clean copies will be created with a “_Clean” suffix. It is a totally free tool to help community to prevent long import times and waste of database space. Here is the link for download : https://www.dropbox.com/s/qrvxchtwsvs5778/POBS%20Cleaner.rar?dl=0
Thank you Ahmet Kochan. Great help indeed.
Ahmet, thanks for sharing! Can’t wait to test it out.
Michael
Thank you very much!! Perfect tool! 5 stars!!
The way I used was a macro in Notepad++. But yours is awesome!!!
Great Tool Ahmet Thanks
Thanks Ahmet !!
Very Useful
Dear Ahmet Kochan,
You are the man!
Very simple & effective tool.
Thanks,
PDA
How about just deleting the POBS table in XER File Parser? Then rebuild the XER.
Would this work?
Regards
Davo
Davo, that works fine as long as you get the XER File Parser to work. I know many users have difficulties with it.
Michael
I recently heard from colleagues that this issue has been fixed in the new release of P6v15. Do you know if this is true?
Regards,
John
Hi John, I just confirmed from Oracle this is fixed in P6 v15.1.5.
Sad to report that it is not fixed in the latest. I am using 15.2 and my normal export is usually 1.16MB… this week it was 16.6 MB.
I had 65000+ lines of POBS data.
I used AHMET’s fix though and it worked like a charm.
Mr. Michael,
Thank you very much for your kind information.
I’ve got bigggg problem about this issue till yesterday. It took 3 hours to import a schedule.
Thank you, again.
Thank you Michael. Big help on finding this post when the import file kept crashing with over 11,000 POBS in a 50 activity schedule!
Superb Michael it indeed saved a lot of blushes
Fantastic! Thanks so much!
For those just using Notepad with many lines (to avoid scrolling):
Mine had about 300,000 lines. Since I don’t have Notepad++, I simply:
1) CTRL+F the POBS %T
2) Find next
3) Highlight the row above the next %T
4) Click the SCROLL bar and drag it all the way to the top
5) Find that %T next to the POBS (should be easy to locate, since it’ll be really close to the top
6) Hold SHIFT, highlight to the %T next to the POBS
7) Delete, and hit CTRL + S to save,
8) Import your file
This saved me about…. well…. god knows how long. 300,000 lines is not fun to scroll through. Just figured i’d offer a quick tip for those with limited resources
Dear Michael,
I used to import a project in about 45 minutes whose file size is about 31 MB. Then I cleaned the file using your method and I shaved off 53,200 lines from a total of 207,000 lines, which reduced the file size to around 23 MB.
Now, I am importing in about 20 minutes.
Is there anyway for me to reduce the file size further and hence reducing the importing time?
The schedule is resource loaded.
James – look above for a comment from Ahmet Kochan – he’s got a nice little app for free that will clean up POBS in an xer file. 20 minutes still seems very long. Try his app.
I have downloaded Ahmet’s .rar file but I do not know how to use it. What is the actual process to clean the xer. If someone could outline the step by step process I’d be very grateful.
Thanks
Thanks Ahmet Great Tool indeed
Great Tool! Thanks Ahmet!!
Hello:
I tested Ahmet’s tool and found a problem with it. I wrote my own little tool in C, and it simply copies byte for byte the file, skipping the offending information.
Ahmet’s tool apparently reads characters, and those characters are interpreted according to local language, or character sets, or whatever. You can see the problem in the line corresponding to currencies, in which the pound sign is changed and altered. It also mangles characters with accents, and imagine many characters not in the standard english character set.
So a little revision of the tool has to be done.
Nice interface! Mine does not have one it is just a command line C program, but it is so fast that the DOS windows hardly gets noticed. Once is is used, it gets added to the windows explorer context menu, so using it is a breeze for individual files.
Regards
Simon Graffe
Instead of scrolling the normal way with the mouse, once you’ve found the end of the POBS data click with the mouse before the next %T and use the side scroll bar to navigate back to the top of the data in less than a second.
I use notepad to routinely check and clean XERs and this technique takes me less than 10 seconds even on 25MB files!
Thanks for the instructions, this has now become the basis of a work instruction on POBS for our planners. I include a link back to this page. 🙂
Great Comment As usual Mark.
Thanks a lot for such an amazing articles and site.
Happy to hear you’re getting value. Cheers!
Michael
I tried Ahmet’s tool but get runtime errors when importing. Anyone have an alternative besides the Notepad approach? Thanks.
I used the POB Cleaner, Simply fantastic! Thanks for sharing the info.
Big thanks! That saved my butt :))
a Big you’re welcome!
ScheduleCleaner is also featuring POBS cleaning option. See more at https://schedulecleaner.com/
Export to .xml its easy.
For Citrix User, delete rows under RISKTYPE to reduce the file size
Thanks for the tip! Don’t know why that RISKTYPE area is always so large.
Hello,
Is there any way to automate the POBS data as scheduled task?
I don’t know that I would spend time on it. The POBS problem is fixed in later versions of Primavera P6, so I would likely spend my time upgrading instead.
Hi, I have tried to clean POBS by Notepad, my file had 59 MB before and reduce to 22 MB.
Once import the file, i still almost need 2 hours, and the longest time need to import is importing risk category.
Is there any way to solve this issues?
Thanks before
I deleted rows under risk type, and the file reduced to 107 KB
Thank you very much
you’re welcome!
glad you were able to follow the instructions and solve it.
Hi guys!
any of you knows of a tool similar to Ahmet’s (that doesn’t need to be installed in Windows to work) to clean the Risk Types? due to company’s restrictions I can’t install any software.
Many thanks!