Data Loss / Web priorities

Foosball chat / key issues and discussion
Post Reply
User avatar
Martyn
Posts: 1459
Joined: Sat Jan 25, 2003 10:37 pm
Real Name: Martyn Harris
Location: Bristol

Re: Bar Kick open 2009 highs and lows..

Post by Martyn » Tue Sep 15, 2009 11:53 pm

I am slightly concerned.

Joe said :
No tournament ever has everyone saying it was awesome, so sorry if you thought that was the case.
That got me thinking as we ran two cracking tournies at the Bush, so I went to see the feedback we had. All I could reemmber was all good.

Virtually nothing at all. No reports at all, no 2003 who came where etc. Scant information only. See for yourselves.

http://britfoos.com/events/2003/bristol-open 2003

http://britfoos.com/events/2005/bush-open-ii 2005

I PM'ed Tom King to find out what's what and didn't fully understand his reply.

Please someone tell me that by upgrading this board we haven't lost tons and tons of our foosball history.
Insanity: doing the same thing over and over again and expecting different results.
Albert Einstein

User avatar
Oatsey
Posts: 2993
Joined: Sun Feb 16, 2003 8:08 pm
Real Name: Dave Oates
Location: Bristol

Re: Bar Kick open 2009 highs and lows..

Post by Oatsey » Wed Sep 16, 2009 6:52 am

Unfortunately Martyn , my understanding was that the board was hacked into at some point and a lot of data lost. I am sure someone can confirm that.

User avatar
Jonathan may
BFA Committee Observer
Posts: 3817
Joined: Tue Feb 17, 2004 5:21 pm
Real Name: Jonathan May
Location: London
Contact:

Re: Bar Kick open 2009 highs and lows..

Post by Jonathan may » Wed Sep 16, 2009 7:20 am

The data exists but not in a suitable form for reintegration into this forum without a significant amount of work.

At some point the simplest solution would probably be to place it in an "archived forum" somewhere.

If there are any volunteers who might have time and forum experience who want to have a go, get in touch.
--
Manager, TeamGB.

User avatar
tom_k
Site Administrator
Posts: 854
Joined: Thu Jul 17, 2003 9:32 am
Real Name: Tom King
Location: Oxford

Data Loss / Web priorities

Post by tom_k » Wed Sep 16, 2009 8:47 am

Martyn wrote: I PM'ed Tom King to find out what's what and didn't fully understand his reply.
Please someone tell me that by upgrading this board we haven't lost tons and tons of our foosball history.
I didn't fully understand your question :)

There are two current issues re: data at the moment.

1) Old Forum Posts - Before I even started touching the forum, something happened (in 2006?) and all I have is a .sql dump of the data - i.e the absolute raw format of the threads etc. These ideally need to be reintegrated. These posts/threads are in a format which is compatible with V.2.x of the forum, not 3.x which we're now on. So in order to reintegrate the data, I"d need to install a 2.x version of the board, upgrade this second version, and then reintegrate the two datasets. There are various issues with this, not least that the post numbering may cause issues. I am willing to try, it's just a matter of time. As to jon's suggestion, of an archived forum, yes that's do-able, but it's just another PHPBB install to worry about in terms of security/management etc. unless we converted it all into static html (again, do-able).

2) Results & Event Listings - The main events listings on the website is the other issue where I've got a massive gap from 2006 - 2009; we've got the data, but in one huge spreadsheet, output in an non-parsable, e.g inconsistent fashion. This is a real headache. Whilst an obvious amount of effort has gone in to consolidate the data (thanks boris), I need to find a way to auto import this into Drupal, like we did with pre-2006. (and that was thanks to the tireless efforts of Jon). It gets more confusing when you look at results intergrating into Shovo's POY/Results DB - but I'm not going to go there :)

User avatar
tom_k
Site Administrator
Posts: 854
Joined: Thu Jul 17, 2003 9:32 am
Real Name: Tom King
Location: Oxford

Re: Data Loss / Web priorities

Post by tom_k » Wed Sep 16, 2009 9:04 pm

Martyn wrote:

Now that#'s sorted (hopefully), I will add to my comments on missing data.

Where is it??

I know a load of work was done to upgrade the board, big up to TomK for that, but somehow in the spirit of improvements, we seem to have lost huge chunks of history. I would like it back...........otherwise what's the point?.

Or maybe it's just me.

I guess it is, as no one seems to have commented on it - no one seems that bothered.

Bush 1 as I recall was when John W and Herr Morgan surprised a top field and performed heroics........I know much was written, and I'd like to have been able to go back and see it for myself sometime...........wouldn't others?

Sorry to ramble on, but I happen to think our history, who won and when and the feedback comments are really important and if we have lost some or all of it, I hope the board improvemts were worth it.

(mods, feel free to split these lost data posts from the tournament feedbacks)

User avatar
Martyn
Posts: 1459
Joined: Sat Jan 25, 2003 10:37 pm
Real Name: Martyn Harris
Location: Bristol

Re: Data Loss / Web priorities

Post by Martyn » Wed Sep 16, 2009 9:08 pm

Oatsey wrote:
Unfortunately Martyn , my understanding was that the board was hacked into at some point and a lot of data lost. I am sure someone can confirm that.
I think that was 2 board updates ago, Dave - I do recall we did have a major hack issue, but I am pretty sure it was before Bush 1, quite some while ago. Someone will remember, Boris for sure.
Insanity: doing the same thing over and over again and expecting different results.
Albert Einstein

User avatar
tom_k
Site Administrator
Posts: 854
Joined: Thu Jul 17, 2003 9:32 am
Real Name: Tom King
Location: Oxford

Re: Data Loss / Web priorities

Post by tom_k » Wed Sep 16, 2009 9:13 pm

This is probably the 2nd/3rd time it's come up since I initially looked at all this over a year ago.

I'm happy to do it, just need to sort priorities...

"the list" looks a bit like this:

- Reimport Old Forum Data, somehow :)
- Import 2006 - 2009 Data from events into Drupal, somehow :)
- Retheme the forum to have the universal britfoos header
- Retheme the national site to match the other sites
- Possibly integrate forum and Drupal logins for a single sign on solution
- Possibly Integrate PayPal Subscriptions into Drupal / Forum user logins (for BFA recurring membership)
- Possibly integrate PayPal into the events pre-registration system
- Pester people to give me content (for everything)
- Look again for that nice forum plugin saying how many new posts there were (next to new posts)
- Delegate content updating and provision (volunteers, anyone?)

Naturally, there are things on the list which are "would be nice" rather than "give us this now".... :) but at least there is a list!

User avatar
Richard
Posts: 1703
Joined: Thu Jan 02, 2003 3:50 pm
Real Name: Richard Thomson
Location: London

Re: Data Loss / Web priorities

Post by Richard » Fri Sep 18, 2009 10:47 am

I am probably repeating others, but here goes:
- The forum was hacked, and all posts (other than topics and announcements) older than i think 1 month were deleted.I dont know how long it took before we noticed, but suspect it was weeks not days.
- I tried to get the server admins to restore the monthly backups, but they said that whilst the html was restorable, the database backups were empty. Dont ask me what they are paid for.
- The most recent backup i had made was from memory about 4 months out of date. But critically it was a backup and so whilst perhaps 2 months data would be lost, in theory some form of merge was doable. The backup was on the server when i handed over the keys and it sounds like Tom has found it. Incidentally Tom, for the record, could you confirm the date of the backup as recollection can be distorted over time.
- I had quite long discussions with Rob M, where we concluded it was doable but far from straightforward. He did not get the chance to implement it. I did not know how to do it myself, and spent a long time on the computing forum trying to badger others to help, but eventually gave up.
- The forum in those early days had more discussion, more positive vibe and excellent records (like the 6m chair updates). I would love to see it reinstated if it is possible.

If there has been a subsequent or separate loss of data then that is something i am not aware of.

Thanks,
R

User avatar
tom_k
Site Administrator
Posts: 854
Joined: Thu Jul 17, 2003 9:32 am
Real Name: Tom King
Location: Oxford

Re: Data Loss / Web priorities

Post by tom_k » Fri Sep 18, 2009 11:07 am

Richard wrote: - I tried to get the server admins to restore the monthly backups, but they said that whilst the html was restorable, the database backups were empty. Dont ask me what they are paid for.
Pleasingly, we've now left those incompetents behind :)
I have never, ever been so shocked at the lamentable hosting that was [that company].
Richard wrote: - The most recent backup i had made was from memory about 4 months out of date. But critically it was a backup and so whilst perhaps 2 months data would be lost, in theory some form of merge was doable. The backup was on the server when i handed over the keys and it sounds like Tom has found it. Incidentally Tom, for the record, could you confirm the date of the backup as recollection can be distorted over time.
i'll dig out what I've got and report back
Richard wrote: If there has been a subsequent or separate loss of data then that is something i am not aware of.
Not to my knowledge. :)

User avatar
tom_k
Site Administrator
Posts: 854
Joined: Thu Jul 17, 2003 9:32 am
Real Name: Tom King
Location: Oxford

Re: Data Loss / Web priorities

Post by tom_k » Fri Sep 18, 2009 11:34 am

Ok, the SQL dump I've got it from 31-05-2005 12:33:01 GMT.

The first post in the SQL dump appears to be Wed, 15 Jan 2003 18:59:34 GMT (if my Unix timestamp conversion methods are correct :P)
The first post in our current forum is an indentical postID and timestamp.

Do we know what time period the missing data is from - to?

User avatar
Jonathan may
BFA Committee Observer
Posts: 3817
Joined: Tue Feb 17, 2004 5:21 pm
Real Name: Jonathan May
Location: London
Contact:

Re: Data Loss / Web priorities

Post by Jonathan may » Fri Sep 18, 2009 11:46 am

Tom_K wrote:Ok, the SQL dump I've got it from 31-05-2005 12:33:01 GMT.

The first post in the SQL dump appears to be Wed, 15 Jan 2003 18:59:34 GMT (if my Unix timestamp conversion methods are correct :P)
The first post in our current forum is an indentical postID and timestamp.

Do we know what time period the missing data is from - to?
Is there not a way to get the current form to spit out "total monthly posts" going back to 2003.

This will almost immediately give us a rough indication of the missing posts and will let us narrow down exactly the dates in question.

I can think of other methods but not without knowing the structure of the data. I also "miss" these posts as they cover the early Cambridge events I ran too, I think.
--
Manager, TeamGB.

User avatar
AK47
Posts: 469
Joined: Tue Dec 19, 2006 12:13 am
Real Name: Antony Kay
Location: Bolton/Manchester

Re: Data Loss / Web priorities

Post by AK47 » Fri Sep 18, 2009 1:17 pm

Tom_K wrote: - Look again for that nice forum plugin saying how many new posts there were (next to new posts)
So glad to see you haven't forgot, Tom. I will never doubt you again.
Every Friday at Rileys Manchester on Deansgate. PM me for more info.

User avatar
robmoss2k
Posts: 1682
Joined: Mon Feb 10, 2003 3:36 pm
Real Name: Robert Moss
Location: Bolton
Contact:

Re: Data Loss / Web priorities

Post by robmoss2k » Mon Sep 21, 2009 9:42 am

Richard wrote:- I had quite long discussions with Rob M, where we concluded it was doable but far from straightforward. He did not get the chance to implement it. I did not know how to do it myself, and spent a long time on the computing forum trying to badger others to help, but eventually gave up.
I had a few goes but never got anything working and gave up in the end. It was always the number ranges that clashed with each other and when I tried to fudge things so they didn't clash I could never get the forum to work. Someone with a better understanding of the innards of phpBB2 would stand a better chance, it certainly looks possible.
Image

Post Reply