Server for Data module

[Kirill]

Background

We run our server log processing (2.0) currently on a server a data center, where we will reach our hardware limits soon.

What to change

We need a new server where we can deploy our processes. The server should be secured and in one of the data centers as well. Besides this we need sudo rights to establish our processes ourselves.

Our absolute minimum hardware requirements:

Hard drive: 4TB
RAM: 32GB.
CPU: 4 physical

Our actual requirements:

Hard drive: 8TB
RAM: 64GB.
CPU: 8 physical

If you want to make us happy and get a pizza:

Hard drive: 8TB
RAM: 128GB.
CPU: 8 physical

Software requirement:

Ubuntu 14.04

Currently our process and adhoc queries may take a lot of RAM. We already experienced restrictions of parallel working (and parallel processing) in our work. More RAM would help us speed up the process (which is very crucial, since we have a lot of reprocessing recently) and we could improve our workflow, by not blocking each other adhoc tasks.

[matze]

Thank you for notifying us about this ticket. Before marking this one as ready, a few points need to be addressed:

• This is not really about a "production environment" server. It would be a server "in one of our data-centers", able to "access the production environment". That's details, of course, but may lead to misinterpretation if not addressed.
• Please do not set the priority of Infrastructure tickets, it has no effect for you anyway. You may, however, address the urgency or a possible due date in the description or comments, i.e. when you expect your current workflow to become impossible without this new entity, or if another issue is blocked by it or else.
• What exactly to you need sudo(8) privileges for, which commands in particular? There won't be a "wildcard" on a server officially administered, so we either nail it down to a set of particular commands, or you may request a server that is not part of the Infrastructure module's responsibilities.

I've partially updated the ticket description according to these points, please take over and adjust the remaining bits.

As a side-note: I expect pizza irregardless the amount of RAM you get, in fact I expect a pizza every day from now on. And at some point we have to address the circumstance that your processes require way more resources than they process anyway.

[Kirill]

What exactly to you need sudo(8) privileges for, which commands in particular? There won't be a "wildcard" on a server officially administered, so we either nail it down to a set of particular commands, or you may request a server that is not part of the Infrastructure module's responsibilities.

We need sudo for apt-get, editing of global config files, e.g /etc/hosts or /etc/apt/sources.list and switching between users (postgres, datamaster user).

Everything else is clear.

[matze]

• invoking apt-get: not possible on an official infrastructure server
• editing /etc/hosts or /etc/apt/sources.list: not possible on an official infrastructure server
• switching between particular users: no problem

[fred]

At the last Infrastructure-Data-Meeting we decided that this server should actually be provided as an in-house ressource by Office-IT, because then we can give the data module all the local admin privileges on the machine they wish for in this ticket. We will still be able to secure the data on the server by limiting the network traffic to and from the machine. This is something we cannot do at the external datacenter.

[fred]

The server hardware has been ordered (Order Ticket #319).

[fred]

Server has been installed with Ubuntu 14.04 LTS with Software RAID, LVM and disk encryption following the suggestions in this howto: ​http://blog.asiantuntijakaveri.fi/2014/12/headless-ubuntu-1404-server-with-full.html

So far, only the first two harddisks have been setup.
The other two disks still need to be added into the RAID array, so we have a RAID10 setup for a good balance between performance and data safety.

[fred]

The other two disks have been added to the RAID array.
But since it was not possible to extend the existing RAID10 array, the same functionality has now been achieved using LVM.