Gruppo Meteo/HowTo/server

Da raspibo.
Jump to navigation Jump to search

Installazione

repo rmap

  • copiare in /etc/yum.repos.d/ il file dal link
http://rmapv.rmap.cc/repo/rmap/rmapcentos.repo

oppure 

http://rmapv.rmap.cc/repo/rmap/rmap.repo
  • installare rmap con le dipendenze
yum groupinstall rmap
  • installare monit
yum install monit
  • configurare monit
/etc/monit.d/rmap
/etc/monitrc

tmpfiles

  • creare il file rmap.conf in /etc/tmpfiles.d
d /run/wsgirmap 0755 rmap rmap -
d /var/run/rmap 0755 rmap rmap -
d /var/run/httpd 0755 rmap rmap -


RMAP

  • Installare il pacchetto MySQL-python
  • Installare Apache HTTP Server: pacchetti httpd mod_wsgi
  • Installare MariaDB
  • Installare rmap.noarc
  • Inizializzare il DB di rmap (file di configurazione in /etc/rmap )
rmapctrl --syncdb
 rmap:pamr
  • Modificare /etc/httpd/conf/httpd.conf
User rmap
Group rmap
  • aggiungere in /etc/httpd/conf.d
 arkiweb.conf
 graphite-web.conf
 rmap.conf
  • agiungere la directory per i socket WSGI
mkdir /run/wsgirmap/
chown rmap:rmap /run/wsgirmap/
  • Cambiare le impostazioni di invio SMTP
/etc/rmap/rmap-site.cfg
  • Copiare http2mqtt in /var/www/html/http2mqtt
 http://sourceforge.net/p/r-map/code/HEAD/tree/trunk/php/

mariaDB

yum install mariadb-server mariadb-libs
  • copiare in /etc/my.cnf.d il file rmap.cnf
[mysqld]

datadir = /rmap/mysql
skip-networking
server-id       = 1
default-storage-engine = InnoDB
innodb_file_per_table
innodb_data_home_dir = /rmap/mysql
innodb_data_file_path = ibdata1:10M:autoextend
innodb_log_group_home_dir = /var/lib/mysql
  • creare la directory /rmap/mysql
mkdir -p /rmap/mysql
chown mysql:mysql /rmap/mysql
  • far partire il DB server
systemctl start mariadb.service

mysql_secure_installation
  • creare il DB
mysql -u root -p
create database rmap;
CREATE USER 'rmap'@'%' IDENTIFIED BY 'rmap';
GRANT ALL PRIVILEGES ON rmap.* TO 'rmap'@'%' with GRANT option;

CREATE USER 'rmapadmin'@'%' IDENTIFIED BY 'rmapadmin';
create database rmapadmin;
GRANT ALL PRIVILEGES ON rmapadmin.* TO 'rmapadmin'@'%' with GRANT option;

FLUSH PRIVILEGES;

rabbitmq-server

  • è necessario installare due pacchetti

python-pika per i client

rabbitmq-server per il server 

rpm --import https ://www.rabbitmq.com/rabbitmq-signing-key-public.asc
yum install http ://www.rabbitmq.com/releases/erlang/erlang-18.3-1.el7.centos.x86_64.rpm
yum install  http ://www.rabbitmq.com/releases/rabbitmq-server/v3.6.1/rabbitmq-server-3.6.1-1.noarch.rpm
  • attivazione server e plugin
chkconfig rabbitmq-server on
service rabbitmq-server restart

rabbitmq-plugins enable rabbitmq_management
rabbitmq-plugins enable rabbitmq_shovel
rabbitmq-plugins enable rabbitmq_shovel_management
  • cambio password amministrazione:

rabbitmqctl change_password guest 123456

  • verifica risoluzione nome; nel caso fare:

The next thing to do is add the hostname’s short names in your /etc/hosts file. Why? Because RabbitMQ does not like FQDN (Attempting to do this will result in the following error: ** System NOT running to use fully qualified hostnames **).

  • gestione porta TCP

la porta dedicata a AMQP è la 15672 grantire l'accesso dove e come necessario

  • gestione porta interfaccia grafica management

The web UI is located at: http://server-name:15672/ on a fresh installation the user "guest" is created with password "guest", ma noi la password l'abbiamo cambiata. attenzione ci si collega di default solo da localhost se non va bene bisogna cambiare la conf di loopback_users in /etc/rabbitmq/rabbitmq.config 

[
{rabbit,
         [
         %% Uncomment the following line if you want to allow access to the
         %% guest user from anywhere on the network.
           {loopback_users, []},
  • configurazione

Dal menù "Overview" selezionare la voce "Import / export definitions" e "upload broker definition" 

https ://github.com/r-map/rmap/blob/master/rabbitmq/rabbit_server.json


  • tuning

ecco solo alcuni suggerimenti:


file rabbitmq.config

disk_free_limit

Disk free space limit of the partition on which RabbitMQ is storing data. When available disk space falls below this limit, flow control is triggered. The value may be set relative to the total amount of RAM (e.g. {mem_relative, 1.0}). The value may also be set to an integer number of bytes. By default free disk space must exceed 50MB. See the memory-based flow control documentation.

Default: 50000000


Controlling System Limits on Linux

The main setting that needs adjustment is the max number of open files, also known as ulimit -n. The default value on many operating systems is too low for a messaging broker (eg. 1024 on several Linux distributions). We recommend allowing for at least 65536 file descriptors for user rabbitmq in production environments. 4096 should be sufficient for most development workloads.

file /etc/security/limits.d/50-rabbitmq.conf 

#
#<domain>      <type>  <item>         <value>
#

rabbitmq         soft    nofile          4096
rabbitmq         hard    nofile          65536


rabbitmq_auth_backend_http

fare riferimento a https://github.com/simonmacmullen/rabbitmq-auth-backend-http

Sacricare il plugin da qui http://www.rabbitmq.com/community-plugins.html e metterlo in: 

/usr/lib/rabbitmq/lib/rabbitmq_server-3.3.5/plugins/

attivarlo con: 

rabbitmq-plugins enable rabbitmq_auth_backend_http

aggiungere in /etc/rabbitmq/rabbitmq.config 

 {rabbit, [{auth_backends, [rabbit_auth_backend_http]}]},

Risulterà, per esempio: 

[
{rabbit,
         [
         %% Uncomment the following line if you want to allow access to the
         %% guest user from anywhere on the network.
           {loopback_users, []},

           {auth_backends, [rabbit_auth_backend_internal, rabbit_auth_backend_http]}
         ]
},

{rabbitmq_auth_backend_http,
         [
           {user_path,     "http://localhost/auth/user"},
           {vhost_path,    "http://localhost/auth/vhost"},
           {resource_path, "http://localhost/auth/resource"}
         ]

}
].

Riavviare il servizio 

service rabbitmq-server restart

showel configuration

in /etc/sudoers.d/rmap 

Defaults:rmap !requiretty
rmap ALL= NOPASSWD: /sbin/rabbitmqctl

DB-All.e

  • installare il pacchetto mysql-connector-odbc
  • aggiungere il DSN in /etc/odbc.ini


[rmap]
Description     = MySQL rmap database
Trace       = Off
TraceFile   = stderr
Driver      = MySQL
SERVER      = localhost
USER        = rmap
PASSWORD    = rmap
PORT        = 3306
DATABASE    = rmap
  • inizializza il DB
dbadb wipe --dsn=rmap


arkimet

  • dnf install arkimet
  • aggiungere alias arkiserver al hostname in /etc/hosts
  • creare utente rmap
useradd -r -s /sbin/nologin rmap
  • TODO aggiungere dal repo svn il file
 /etc/arkimet/scan-bufr/generic.lua
  • editare /etc/sysconfig/arkimet
# Configuration for starting an instance of arki-server
# This is going to be sourced by systemd thus no variable expansion is performed

# Directory where the log files are written (must exist)
LOGDIR="/var/log/arkimet"

# Options related to networking
NETWORK_OPTIONS="--url=http://arkiserver:8090 --port=8090"

# Any other option
OTHER_OPTIONS="--quiet"

# Configuration file to use, normally generated by arki-mergeconf. If this
# variable is left empty, the server is not loaded.
DATASET_CONFIG="/rmap/arkimet/arkimet.conf"
  • change in /usr/lib/systemd/system/arkimet.service
User=rmap
  • creare la directory per i dati
mkdir /rmap/arkimet
chown rmap:rmap /rmap/arkimet
  • configurare arkimet scompattando la struttura del file

arkimetconf.tgz 

 chown -R rmap:rmap /rmap/arkimet

arkiweb

  • dnf install arkiweb
  • in /etc/httpd/conf.d creare il file arkiweb.conf
ScriptAlias /services/arkiweb/ /usr/lib/arkiweb/
<Directory "/usr/lib/arkiweb">
        AllowOverride None
        Options +ExecCGI

    	Order allow,deny
    	Allow from all

        # ARKIWEB_CONFIG is mandatory!
        SetEnv ARKIWEB_CONFIG /rmap/arkimet/arkiweb.config
        

	Require all granted

        # Authentication (optional)
        #
        # Basic authentication example:
        # SetEnv ARKIWEB_RESTRICT REMOTE_USER
        # AuthType Basic
        # AuthUserFile /etc/arkiweb.passwords
        # require valid-user
</Directory>

Alias /arkiwebjs/ /usr/share/arkiweb/public/
<Directory "/usr/share/arkiweb/public">
	   #Require all granted
	   AllowOverride None
</Directory>
  • creare il file /rmap/arkimet/arkiweb.config

[meteonetwork]
bounding = POLYGON ((12.6577099999999998 43.8649699999999996, 10.5704300000000000 44.3001199999999997, 9.8880599999999994 44.5129299999999972, 9.4983599999999999 44.6443500000000029, 9.4859799999999996 44.7079000000000022, 9.4444999999999997 44.9392799999999966, 9.4909800000000004 45.0587200000000010, 9.6209699999999998 45.0564999999999998, 12.0983400000000003 44.9064100000000010, 12.2681299999999993 44.8038799999999995, 12.6788299999999996 43.9913799999999995, 12.6686999999999994 43.8718500000000020, 12.6577099999999998 43.8649699999999996))
filter = product: BUFR:t=mnw
index = reftime, area, product, origin, proddef
name = meteonetwork
path = /rmap/arkimet/meteonetwork
postprocess = json
replace = yes
step = daily
type = ondisk2
unique = reftime, area, product, origin, proddef

[opendata-er]
bounding = POLYGON ((12.1362299999999994 43.6922300000000021, 9.2420700000000000 44.4809800000000024, 9.1545400000000008 44.5148500000000027, 9.2314900000000009 44.8656700000000015, 9.5297699999999992 45.0566800000000001, 9.7055399999999992 45.0605199999999968, 12.1221499999999995 44.9429000000000016, 12.2503600000000006 44.8128699999999967, 12.7393999999999998 43.9584699999999984, 12.6429399999999994 43.9118199999999987, 12.1362299999999994 43.6922300000000021))
filter = product: BUFR:t=rer or BUFR:t=simc or BUFR:t=urbane
index = reftime, area, product, origin, proddef
name = opendata-er
path = /rmap/arkimet/opendata-er
postprocess = json
replace = yes
step = daily
type = ondisk2
unique = reftime, area, product, origin, proddef

[rmap]
bounding = POLYGON ((12.1362299999999994 43.6922300000000021, 9.2420700000000000 44.4809800000000024, 9.1545400000000008 44.5148500000000027, 9.2314900000000009 44.8656700000000015, 9.5297699999999992 45.0566800000000001, 11.1234500000000001 45.1234499999999983, 12.1221499999999995 44.9429000000000016, 12.2503600000000006 44.8128699999999967, 12.7393999999999998 43.9584699999999984, 12.6429399999999994 43.9118199999999987, 12.1362299999999994 43.6922300000000021))
filter = product: BUFR:t=rmap
index = reftime, area, product, origin, proddef
name = rmap
path = /rmap/arkimet/rmap
postprocess = json
replace = yes
step = daily
type = ondisk2
unique = reftime, area, product, origin, proddef


mosquitto

  • modificare
/etc/mosquitto/mosquitto.conf
/etc/mosquitto/aclfile
/etc/mosquitto/pwfile
/etc/mosquitto/conf.d/rmap.conf
  • per systemd modificare il file /usr/lib/systemd/system/mosquitto.service
[Service]
ExecStart=/usr/sbin/mosquitto  -c /etc/mosquitto/mosquitto.conf 
User=mosquitto

'segnalare BUG'

mosquitto-auth-plug

Riferirsi a: https://github.com/jpmens/mosquitto-auth-plug

copiare i sorgenti in locale

creare config.mk come segue: 

# Select your backends from this list
BACKEND_CDB ?= no
BACKEND_MYSQL ?= no
BACKEND_SQLITE ?= no
BACKEND_REDIS ?= no
BACKEND_POSTGRES ?= no
BACKEND_LDAP ?= no
BACKEND_HTTP ?= yes

# Specify the path to the Mosquitto sources here
MOSQUITTO_SRC =

# Specify the path the OpenSSL here
OPENSSLDIR = /usr

poi: 

make
mkdir /var/lib/mosquitto/plugins
cp auth-plug.so /var/lib/mosquitto/plugins/


creare il file /etc/mosquitto/conf.d/rmap.conf 

auth_plugin /var/lib/mosquitto/plugins/auth-plug.so
auth_opt_backends http
auth_opt_http_hostname localhost
auth_opt_http_ip 127.0.0.1
auth_opt_http_port 80
auth_opt_http_getuser_uri /auth/auth
auth_opt_http_superuser_uri /auth/superuser
auth_opt_http_aclcheck_uri /auth/acl

Modificare il file /etc/mosquitto/aclfile 

# This affects access control for clients with no username.
#topic read $SYS/#
topic read #
topic write test/#

# This only affects clients with username "rmap".
user rmap
topic #

# This affects all clients.
#pattern write $SYS/broker/connection/%c/state

pattern write rmap/%u/#
pattern write report/%u/#
pattern write mobile/%u/#

Aiungere la password del amministratore in /etc/mosquitto/pwfile 

mosquitto_passwd

restart server: 

service mosquitto restart

server ports

* 22         ssh
* 80         webserver
* 443        monit web server
* 1883 8883  MQTT
* 5672       amqp
* 2003       carbon-cache
* 5925       borinud
* 15672      rabbitmq_management
* 8090       arkimet

graphite

modificare: 

/usr/lib/python2.7/site-packages/tagging/forms.py

inserendo: 

fields = "__all__"

dopo: 

model = Tag
  • installare python-django
  • installare graphite-web
  • installare python-carbon
  • editare /etc/carbon/carbon.conf
LINE_RECEIVER_INTERFACE = 127.0.0.1
ENABLE_UDP_LISTENER = True
UDP_RECEIVER_INTERFACE = 127.0.0.1
  • cambiare i permessi di /var/log/graphite-web
chown rmap /var/log/graphite-web
chmod g+w /var/log/graphite-web
  • editare /etc/carbon/storage-schemas.conf
# Schema definitions for Whisper files. Entries are scanned in order,
# and first match wins. This file is scanned for changes every 60 seconds.
#
#  [name]
#  pattern = regex
#  retentions = timePerPoint:timeToStore, timePerPoint:timeToStore, ...

# Carbon's internal metrics. This entry should match what is specified in
# CARBON_METRIC_PREFIX and CARBON_METRIC_INTERVAL settings
[carbon]
pattern = ^carbon\.
retentions = 60:7d

[rmap_10sec_for_10day]
#pattern = .*
pattern = ^rmap\.
retentions = 10s:10d

[report_60sec_for_60day]
#pattern = .*
pattern = ^report\.
retentions = 60s:30d

[default_1min_13months]
pattern = .*
retentions = 1m:395d
  • In /etc/graphite-web/local_settings.py

configurare 

DATABASES = {
    'default': {
        'NAME': 'rmapadmin',
        'ENGINE': 'django.db.backends.mysql',
        'USER': 'rmapadmin',
        'PASSWORD': 'rmapadmin',
        'HOST': '',
        'PORT': ''
    }
}

come in rmap ed eventualmente configurare anche la sezione EMAIL poi: 

/usr/bin/graphite-manage migrate

Al un seccessivo httpd restart i db di rmap e graphite dovrebbero essere unificati.

graphite Fedora 20 fast start

yum install graphite-web python-carbon

python /usr/lib/python2.7/site-packages/graphite/manage.py syncdb

chown -R rmap:rmap /var/lib/graphite-web

service carbon-cache start

python /usr/lib/python2.7/site-packages/graphite/manage.py runserver

python /usr/share/doc/graphite-web/example-client.py

firefox http://127.0.0.1:8000/dashboard/

Start and test graphite interface

Start mqtt2graphite and view the graphite graphs on http://server_ip/

mqtt2graphited run

2014-07-08 23:28:08,051 Starting MQTT2Graphite_5966-localhost.localdomain 

2014-07-08 23:28:08,055 INFO MODE
2014-07-08 23:28:08,084 DEBUG MODE
2014-07-08 23:28:08,213 Connected to broker at localhost as MQTT2Graphite_5966-localhost.localdomain
2014-07-08 23:28:08,253 Subscribing to topic meteo/#
2014-07-08 23:28:08,312 pubblish {'carbon_port': 2003, 'map': {'meteo/#': ('j', None)}, 'sock': <socket._socketobject object at 0xb6a9e9d0>, 'carbon_server': '127.0.0.1'} with id 1
2014-07-08 23:28:08,363 Subscribed: 2 (0,)
2014-07-08 23:28:08,409 CARBONKEY is [meteo.-.1012345_4412345.generic.-_-_-.-_-_-_-.B07030]
2014-07-08 23:28:09,834 meteo.-.1012345_4412345.generic.-_-_-.-_-_-_-.B07030.v 400.000000 1404854888
2014-07-08 23:28:09,863 CARBONKEY is [meteo.-.1137637_4449216.generic.-_-_-.-_-_-_-.B07030]
2014-07-08 23:28:09,926 meteo.-.1137637_4449216.generic.-_-_-.-_-_-_-.B07030.v 400.000000 1404854889
2014-07-08 23:28:11,133 CARBONKEY is [meteo.-.1137637_4449216.generic.254_0_0.103_3000_-_-.B12101]
2014-07-08 23:28:11,155 meteo.-.1137637_4449216.generic.254_0_0.103_3000_-_-.B12101.v 30115.000000 1404854891
2014-07-08 23:28:16,687 CARBONKEY is [meteo.-.1137637_4449216.generic.254_0_0.103_3000_-_-.B12101]
2014-07-08 23:28:16,733 meteo.-.1137637_4449216.generic.254_0_0.103_3000_-_-.B12101.v 30121.000000 1404854896
2014-07-08 23:28:24,895 CARBONKEY is [meteo.-.1137637_4449216.generic.254_0_0.103_3000_-_-.B12101]
2014-07-08 23:28:24,933 meteo.-.1137637_4449216.generic.254_0_0.103_3000_-_-.B12101.v 30121.000000 1404854904


node-red

yum install nodejs npm
npm install -g --unsafe-perm node-red
npm install -g node-red-admin

useradd nodered
passwd nodered -l
usermod -s /bin/false nodered

add in /home/nodered/.node-red/settings.js 

httpRoot: '/nodered'

add /etc/systemd/system/nodered.service 

[Service]
ExecStart= /usr/bin/node-red
Restart=always
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=node-sample
User=nodered
Group=nodered
Environment=NODE_ENV=production

[Install]
WantedBy=multi-user.target

enable apache proxy for node-red

add /etc/httpd/conf.d/nodered.conf 

ProxyPass /nodered http://localhost:1880/nodered
ProxyPassReverse /nodered http://localhost:1880/nodered

ProxyPass /nodered ws://localhost:1880/nodered
ProxyPassReverse /nodered ws://localhost:1880/nodered

secure node-red

node-red-admin hash-pw

in settings.js add 

   adminAuth: {
       type: "credentials",
       users: [{
           username: "admin",
           password: "risultato del comando sopra",
           permissions: "*"
       }],
       default: {
       permissions: "read"
       }
   },
Estratto da "https://www.raspibo.org/wiki/index.php?title=Gruppo_Meteo/HowTo/server&oldid=5995"