Connection scenario with OAI service in M-CORD 4.1

Posted on

Introduce the connection between OAI VNF in CORD

Modify network configuration of OAI BBU, MME, SPGW, HSS

Configuration path:

  • [BBU]
    • /openairinterface5g/targets/PROJECTS/GENERIC-LTE-EPC/CONF
  • [MME]
    • /usr/local/etc/oai/mme.conf
    • /usr/local/etc/oai/freeDiameters/mme_fd.conf
  • [SPGW]
    • /usr/local/etc/oai/spgw.conf
  • [HSS]
    • No need to modify

BBU configuration (Connect to MME and BBU)

/openairinterface5g/targets/PROJECTS/GENERIC-LTE-EPC/CONF rcc.band7.tm1.50PRB.nfapi.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
    ////////// MME parameters:
    mme_ip_address      = ( { ipv4       = "10.0.6.2"; // MME IP
                              ipv6       = "192:168:30::17";
                              active     = "yes";
                              preference = "ipv4";
                            }
                          );

    NETWORK_INTERFACES :
    {

        ENB_INTERFACE_NAME_FOR_S1_MME            = "eth0"; // your BBU config
        ENB_IPV4_ADDRESS_FOR_S1_MME              = "10.0.5.2"; // your BBU ip
        ENB_INTERFACE_NAME_FOR_S1U               = "eth0";
        ENB_IPV4_ADDRESS_FOR_S1U                 = "10.0.5.2";
        ENB_PORT_FOR_S1U                         = 2152; # Spec 2152
    };
  }
);

MACRLCs = (
        {
        num_cc = 1;
        local_s_if_name  = "lo";
        remote_s_address = "127.0.0.1"; // connect to RU 
        local_s_address  = "127.0.0.2"; // local address
        local_s_portc    = 50001;
        remote_s_portc   = 50000;
        local_s_portd    = 50011;
        remote_s_portd   = 50010;
        tr_s_preference = "nfapi";
        tr_n_preference = "local_RRC";
        }
);

MME Configuration (Connect to BBU & HSS & SPGW)

/usr/local/etc/oai/mme.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

...

# ------- MME served GUMMEIs
# MME code DEFAULT  size = 8 bits
# MME GROUP ID size = 16 bits
GUMMEI_LIST = (
     {MCC="208" ; MNC="93"; MME_GID="4" ; MME_CODE="1"; }   # should be same as BBU config
);

# ------- MME served TAIs
# TA (mcc.mnc:tracking area code) DEFAULT = 208.34:1
# max values = 999.999:65535
# maximum of 16 TAIs, comma separated
# !!! Actually use only one PLMN
TAI_LIST = (
     {MCC="208" ; MNC="93";  TAC = "1"; }                                   # should be same as BBU config
);

...

NETWORK_INTERFACES :
{
    # MME binded interface for S1-C or S1-MME  communication (S1AP), can be ethernet interface, virtual ethernet interface, we don't advise wireless interfaces
    MME_INTERFACE_NAME_FOR_S1_MME         = "ens3";                        # MME CONFIG HERE
    MME_IPV4_ADDRESS_FOR_S1_MME           = "10.0.6.2/24";                 # YOUR MME CONFIG HERE

    # MME binded interface for S11 communication (GTPV2-C)
    MME_INTERFACE_NAME_FOR_S11_MME        = "ens3";                           # YOUR NETWORK CONFIG HERE
    MME_IPV4_ADDRESS_FOR_S11_MME          = "10.0.6.2/24";                 # YOUR MME CONFIG HERE
    MME_PORT_FOR_S11_MME                  = 2123;                           # YOUR MME CONFIG HERE
};

...

S-GW_LIST_SELECTION = (
# SPGW IP here
     {ID="tac-lb01.tac-hb00.tac.epc.mnc001.mcc001.3gppnetwork.org" ; SGW_IPV4_ADDRESS_FOR_S11="10.0.8.2/24";} 
);

...

/usr/local/etc/oai/freeDiameter/mme_fd.conf

1
2
# set HSS IP here
ConnectPeer= "hss.openair4G.eur" { ConnectTo = "10.0.7.2"; No_SCTP ; No_IPv6; Prefer_TCP; No_TLS; port = 3868;  realm = "openair4G.eur";};

SPGW configuration (connect to MME)

/usr/local/etc/oai/spgw.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
S-GW :
{
NETWORK_INTERFACES :
{
    # S-GW binded interface for S11 communication (GTPV2-C), if none selected the ITTI message interface is used
    SGW_INTERFACE_NAME_FOR_S11              = "ens5";                         # STRING, interface name, YOUR NETWORK CONFIG HERE
    SGW_IPV4_ADDRESS_FOR_S11                = "10.0.8.2/24";               # SPGW config

    # S-GW binded interface for S1-U communication (GTPV1-U) can be ethernet interface, virtual ethernet interface, we don't advise wireless interfaces
    SGW_INTERFACE_NAME_FOR_S1U_S12_S4_UP    = "ens5";                         # STRING, interface name, YOUR NETWORK CONFIG HERE, USE "lo" if S-GW run on eNB host
    SGW_IPV4_ADDRESS_FOR_S1U_S12_S4_UP      = "10.0.8.2/24";              # SPGW config
    SGW_IPV4_PORT_FOR_S1U_S12_S4_UP         = 2152;                         # INTEGER, port number, PREFER NOT CHANGE UNLESS YOU KNOW WHAT YOU ARE DOING

    # S-GW binded interface for S5 or S8 communication, not implemented, so leave it to none
    SGW_INTERFACE_NAME_FOR_S5_S8_UP         = "none";                       # STRING, interface name, DO NOT CHANGE (NOT IMPLEMENTED YET)
    SGW_IPV4_ADDRESS_FOR_S5_S8_UP           = "0.0.0.0/24";                 # STRING, CIDR, DO NOT CHANGE (NOT IMPLEMENTED YET)
};

Usage

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
ssh head1
head1$ scp ~/.ssh/id_rsa ubuntu@10.1.0.14:~/.ssh
head1$ ssh ubuntu@10.1.0.14

# SSH into Service Instance, reference to nova output
ubuntu@multicolored-jump:~$ ssh ubuntu@172.27.0.2

# This is vHSS Service Instance
ubuntu@vhss:~$ sudo su -
root@vhss:~$ cd ~/openair-cn/SCRIPT/
root@vhss:~/openair-cn/SCRIPT$ ./run_hss

# This is vMME Service Instance
ubuntu@nano:~$ sudo su -
root@nano:~$ cd ~/openair-cn/SCRIPT/
root@nano:~/openair-cn/SCRIPT$ ./run_mme

# This is vSPGW Service Instance
ubuntu@spgw:~$ sudo su -
root@spgw:~$ cd ~/openair-cn/SCRIPT/
root@spgw:~/openair-cn/SCRIPT$ ./run_spgw

Connectivity check between BBU & vEPC

BBU with associated MME

Service dependency graph

Setup a connection between RRU & BBU

first, we should do the nested port foward to access the BBU service

Rule should be add at all node:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
# at cord server

# Forward the packet from CORD server's port 10022 to the head node's port 10022
# enp4s0f0 should be your public network interface 
# 192.168.121.238 is the head node interface
-A PREROUTING -i enp4s0f0 -p tcp -m tcp --dport 10022 -j DNAT --to-destination 192.168.121.238:10022
-A FORWARD -d 192.168.121.238/32 -p tcp -m tcp --dport 10022 -j ACCEPT

# check the rules by iptables

cord@cord:sudo iptables -L

# ssh to head node
cord@cord:ssh head1
vagrant@head1:~$sudo su 

# forward packet from head port's 10022 to compute node's port 10023
-A PREROUTING -i eth0 -p tcp -m tcp --dport 10022 -j DNAT --to-destination 10.1.0.14:10023
-A FORWARD -d 10.1.0.14/32 -p tcp -m tcp --dport 10023 -j ACCEPT

# ssh to compute node
vagrant@head1:~$ssh ubuntu@10.1.0.14

# forward packet form node's port 10023 to oai service's port 22
iptables -A PREROUTING -i mgmtbr -p tcp -m tcp --dport 10023 -j DNAT --to-destination 172.27.0.6:22
-A POSTROUTING -p tcp --dport 22 --dst 172.27.0.6 -j MASQUERADE

# to verify ssh successfully or not in each step 
sudo tcpdump -i <NIC interface> port <PORT number>

The way to update iptables

1
2
3
4
5
6
7
8
9
# note, use iptables-save and iptables-restore would be better to addjust the rules

sudo su

vim iptables.rules

# Add rules and make sure all the policy is what you want

iptables-restore < iptables.rules

The iptables rules that I used for this implementation

CORD Server iptables template
Head node iptables template
Compute node iptables template

for SCTP Forwarding

1
2
-A PREROUTING -i eth0 -p sctp -m sctp --dport 50000 -j DNAT --to-destination 10.1.0.14:50000
-A POSTROUTING -d 10.1.0.14 -o mgmtbr -p sctp -j SNAT --to-source 192.168.121.16

Do sudo iptables -L

At CORD server : 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     tcp  --  anywhere             192.168.121.16       tcp dpt:10022
ACCEPT     sctp --  anywhere             192.168.121.16       sctp dpt:50000
ACCEPT     udp  --  anywhere             192.168.121.16       udp dpt:50010
ACCEPT     all  --  anywhere             anywhere            
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable
ACCEPT     all  --  anywhere             10.100.198.0/24      ctstate RELATED,ESTABLISHED
ACCEPT     all  --  10.100.198.0/24      anywhere            
ACCEPT     all  --  anywhere             anywhere            
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable
ACCEPT     all  --  anywhere             192.168.121.0/24     ctstate RELATED,ESTABLISHED
ACCEPT     all  --  192.168.121.0/24     anywhere            
ACCEPT     all  --  anywhere             anywhere            
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable
ACCEPT     all  --  anywhere             192.168.122.0/24     ctstate RELATED,ESTABLISHED
ACCEPT     all  --  192.168.122.0/24     anywhere            
ACCEPT     all  --  anywhere             anywhere            
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable

At head node : 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     tcp  --  anywhere             10-1-0-14.cord.lab   tcp dpt:10023
ACCEPT     sctp --  anywhere             10-1-0-14.cord.lab   sctp dpt:50001
ACCEPT     udp  --  anywhere             portly-eye.cord.lab  udp dpt:50011
DOCKER-USER  all  --  anywhere             anywhere            
DOCKER-ISOLATION  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere

Successfully implement the SSH tunnel

Configure oaibbu with public network

Add Network definition into ==mcord-oai-services.yml.j2==

1
2
3
4
5
6
7
8
9
10

oaibbu_public_network:
  type: tosca.nodes.NetworkSlice
  requirements:
    - network:
        node: public
        relationship: tosca.relationships.BelongsToOne
    - slice:
        node: {{ site_name }}_oaibbu
        relationship: tosca.relationships.BelongsToOne

After rebooting

1
2
3
4
5
6
7
8
9
10
11
vagrant@head1:~$ source /opt/cord_profile/admin-openrc.sh
vagrant@head1:~$ nova list --all-tenants
+--------------------------------------+-------------------+--------+------------+-------------+-----------------------------------------------------------------+
| ID                                   | Name              | Status | Task State | Power State | Networks                                                        |
+--------------------------------------+-------------------+--------+------------+-------------+-----------------------------------------------------------------+
| 0b2aebaf-f5de-421e-8d09-bf27619d33bc | mysite_oaibbu1-5  | ACTIVE | -          | Running     | management=172.27.0.6; public=10.8.1.4; oaibbu_network=10.0.4.2 |
| 64ece792-9e7f-4125-8fd5-9e5cb90234cd | mysite_oaispgw1-1 | ACTIVE | -          | Running     | management=172.27.0.2; public=10.8.1.2; vspgw_network=10.0.8.2  |
| 958ad833-205e-489a-99a8-cf637e813366 | mysite_oaispgw1-4 | ACTIVE | -          | Running     | management=172.27.0.4; public=10.8.1.3; vspgw_network=10.0.8.3  |
| 9d2f47b6-e4e3-4f77-812b-878749185b1a | mysite_vhss1-3    | ACTIVE | -          | Running     | management=172.27.0.5; vhss_network=10.0.7.2                    |
| ed42a403-25c7-4dfd-a3c4-f188997b85ba | mysite_vmme1-2    | ACTIVE | -          | Running     | management=172.27.0.3; vmme_network=10.0.6.2                    |
+--------------------------------------+-------------------+--------+------------+-------------+-----------------------------------------------------------------+

Connection scenario BBU & RRU

BBU configuration

==rcc.band7.tm1.50PRB.nfapi.conf==

1
2
3
4
5
6
7
8
9
10
11
12
13
14
MACRLCs = (
	{
	num_cc = 1;
      	local_s_if_name  = "eth0";			  
      	remote_s_address = "140.118.123.218"; # RRU Physical machine IP
    	local_s_address  = "10.0.4.2"; # Private IP in CORD
    	local_s_portc    = 50000;	
    	remote_s_portc   = 50001;
    	local_s_portd    = 50010;	
    	remote_s_portd   = 50011;
	tr_s_preference = "nfapi";
	tr_n_preference = "local_RRC";
        }  
);

RRU Configuration

==oaiL1.nfapi.usrpb210.conf==

1
2
3
4
5
6
7
8
9
10
11
12
13
L1s = (
    	{
	num_cc = 1;
	tr_n_preference = "nfapi";
      	local_n_if_name  = "eth0";			  
      	remote_n_address = "140.118.122.109";
    	local_n_address  = "140.118.123.218";  # RRU Physical machine IP
    	local_n_portc    = 50001;	
        remote_n_portc   = 50000;
        local_n_portd    = 50011;
        remote_n_portd   = 50010;
        }  
);

Forwarding packet path (STCP & UDP):
CORD Server=> Control node => Compute node => OAI bbu public IP
140.118.122.109 => 192.168.121.1 => 10.1.0.14 = 10.0.4.2

BBU Connect to RRU in physical machine

branch: be8d32d5

Automation scripts

==Command for automation ip setting==

1
ssh <hostname>@<IP> 'bash -s' < test.sh

==test.sh==

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
#!/bin/bash

# Server iptables configuration 
echo start Iptables configuration for server

git clone https://gist.github.com/7d6c9b8d0f2e8389e07f328689340813.git ip
cd ip/
# adapt iptables rule
sudo iptables-restore < Iptables_cord_server.rules

# head1 iptables configuration
ssh head1
git clone https://gist.github.com/cbfbee6e9413cd48bdc8124b3402c0ab.git ip
cd ip/
# adapt iptables rule
sudo iptables-restore < Iptables_head_node.rules

# compute node iptables configuration
ssh ubuntu@10.1.0.14
git clone https://gist.github.com/8f10171c052f88b94b39daf7b4017e74.git ip
cd ip/
sudo iptables-restore < Iptables_compute_node.rules