Tunneler 0.9.0 - Error - nil pagination in response to GET /services

I installed v0.9.0 tunneler and getting the following error everytime the service poller runs, I guess (15s):
[ 62.723] ERROR github.com/netfoundry/ziti-edge/tunnel/intercept.ServicePoller: failed to get ziti services: nil pagination in response to GET /services

Make sure controller version matches tunneler

./ziti-tunnel version
v0.9.0
ziti-controller version
v0.9.0
They look the same to me

Is there an error on the controller side as well?

#033[34m[154465.343]#033[39m #033[31m ERROR#033[39m #033[36mgithub.com/netfoundry/ziti-edge/controller/response.(*RequestResponderImpl).RespondWithError#033[39m: #033[96m{cause=[service with id dcd3d91d-9af6-437e-9ea5-7e78bd831a5b not found]} #033[39munhandled error: service with id dcd3d91d-9af6-437e-9ea5-7e78bd831a5b not found

Thank you, that’s helpful. Can you share what role attributes the identity in question has, as well as any relevant service policies and role attributes on that service?

{
  "meta": {
    "filterableFields": [
      "id",
      "createdAt",
      "updatedAt",
      "name",
      "fingerprint",
      "isVerified",
      "enrollmentToken",
      "enrollmentCreatedAt",
      "enrollmentExpiresAt"
    ],
    "pagination": {
      "limit": 10,
      "offset": 0,
      "totalCount": 1
    }
  },
  "data": [
    {
      "id": "d1066752-e348-4cb6-bdb5-6eeb506b11b3",
      "createdAt": "2020-02-12T21:50:35.7849366Z",
      "updatedAt": "2020-02-12T22:21:26.181338719Z",
      "_links": {
        "edge-router-policies": {
          "href": "./edge-routers/d1066752-e348-4cb6-bdb5-6eeb506b11b3/edge-routers"
        },
        "self": {
          "href": "./edge-routers/d1066752-e348-4cb6-bdb5-6eeb506b11b3"
        }
      },
      "tags": {},
      "name": "ziti-gw01",
      "fingerprint": "7B:D7:35:B9:5C:0F:39:6C:59:8C:E8:88:B5:C1:B7:9F:11:8B:0A:F8",
      "roleAttributes": [
        "dariusz"
      ],
      "isVerified": true,
      "isOnline": true,
      "enrollmentToken": null,
      "enrollmentJwt": null,
      "enrollmentCreatedAt": null,
      "enrollmentExpiresAt": null,
      "hostname": "zedeapptest01.centralus.cloudapp.azure.com:3022",
      "supportedProtocols": {
        "tls": "tls://zedeapptest01.centralus.cloudapp.azure.com:3022"
      }
    }
  ]
}
{
    "id": "3b1d5f9b-d4e5-4e35-9b80-b58d9c9b00dc",
    "createdAt": "2020-02-13T17:56:50.099314709Z",
    "updatedAt": "2020-02-13T18:58:45.098870003Z",
    "_links": {
      "edge-router-policies": {
        "href": "./identities/3b1d5f9b-d4e5-4e35-9b80-b58d9c9b00dc/edge-routers"
      },
      "self": {
        "href": "./identities/3b1d5f9b-d4e5-4e35-9b80-b58d9c9b00dc"
      },
      "service-policies": {
        "href": "./identities/3b1d5f9b-d4e5-4e35-9b80-b58d9c9b00dc/identities"
      }
    },
    "tags": {},
    "name": "dariusz-second",
    "type": {
      "entity": "identity-types",
      "id": "5b53fb49-51b1-4a87-a4e4-edda9716a970",
      "name": "Device",
      "_links": {
        "self": {
          "href": "./identity-types/5b53fb49-51b1-4a87-a4e4-edda9716a970"
        }
      }
    },
    "isDefaultAdmin": false,
    "isAdmin": false,
    "authenticators": {},
    "enrollment": {},
    "roleAttributes": [
      "dariusz1"
    ]
  }
[
  {
    "id": "dcd3d91d-9af6-437e-9ea5-7e78bd831a5b",
    "createdAt": "2020-02-12T22:22:17.902229017Z",
    "updatedAt": "2020-02-13T18:23:42.127592555Z",
    "_links": {
      "self": {
        "href": "./services/dcd3d91d-9af6-437e-9ea5-7e78bd831a5b"
      },
      "service-edge-router-policies": {
        "href": "./services/dcd3d91d-9af6-437e-9ea5-7e78bd831a5b/service-edge-router-policies"
      },
      "service-policies": {
        "href": "./services/dcd3d91d-9af6-437e-9ea5-7e78bd831a5b/identities"
      }
    },
    "tags": {},
    "name": "dariusz01",
    "endpointAddress": "tcp:127.0.0.1:22",
    "egressRouter": "d1066752-e348-4cb6-bdb5-6eeb506b11b3",
    "roleAttributes": [
      "dariusz"
    ],
    "permissions": [
      "Bind",
      "Dial"
    ],
    "configs": null,
    "config": {}
  }
]
[
  {
"id": "e711f1c1-17c0-4dc7-be8c-c7768d8880ea",
"createdAt": "2020-02-13T18:12:34.415029176Z",
"updatedAt": "2020-02-13T18:12:34.415029176Z",
"_links": {
  "edge-routers": {
    "href": "./edge-router-policies/e711f1c1-17c0-4dc7-be8c-c7768d8880ea/edge-routers"
  },
  "identities": {
    "href": "./edge-router-policies/e711f1c1-17c0-4dc7-be8c-c7768d8880ea/identities"
  },
  "self": {
    "href": "./edge-router-policies/e711f1c1-17c0-4dc7-be8c-c7768d8880ea"
  }
},
"tags": {},
"name": "EdgeRouterPolicy1",
"semantic": "AnyOf",
"edgeRouterRoles": [
  "#dariusz"
],
"identityRoles": null
  }
]

also, getting this error when trying to add an identity to a service policy through ZAC:
{“error”:{“args”:{“cause”:{},“urlVars”:{}},“cause”:{},“causeMessage”:“duplicate value ‘ServicePolicy1’ in unique index on servicePolicies store”,“code”:“UNHANDLED”,“message”:“An unhandled error occurred”,“requestId”:“d661cd77-c1e4-44b3-a93e-033aaab9f2a3”},“meta”:{“apiEnrolmentVersion”:“0.0.1”,“apiVersion”:“0.0.1”}}
will try through API to see if I get the same error

I see everything but the service policy, can you post that one, please?

[
  {
    "id": "abfffd81-2299-4d72-8eb1-02c1843ae466",
    "createdAt": "2020-02-13T20:25:38.203442946Z",
    "updatedAt": "2020-02-13T20:25:38.203442946Z",
    "_links": {
      "identities": {
        "href": "./service-policies/abfffd81-2299-4d72-8eb1-02c1843ae466/identities"
      },
      "self": {
        "href": "./service-policies/abfffd81-2299-4d72-8eb1-02c1843ae466"
      },
      "services": {
        "href": "./service-policies/abfffd81-2299-4d72-8eb1-02c1843ae466/services"
      }
    },
    "tags": {},
    "name": "ServicePolicy1",
    "type": "Bind",
    "semantic": "AnyOf",
    "serviceRoles": [
      "#dariusz"
    ],
    "identityRoles": [
      "#dariusz"
    ]
  }
]

I’m guessing you want type “Dial” for that policy, unless you’re trying to host that service? Either way, you shouldn’t get that failure. I’m going to see if I can replicate, but in the meantime do you want to try with a Dial policy?

Dial is better, but now I get the tcp reset.

[   2.457]    INFO github.com/netfoundry/ziti-edge/tunnel/intercept.updateServices: starting tunnel for newly available service dariusz01
[   2.462]    INFO github.com/netfoundry/ziti-edge/tunnel/intercept/tproxy.(*tProxyInterceptor).intercept: Adding rule iptables -t mangle -A NF-INTERCEPT [-m comment --comment dariusz01 -d 3.3.3.3/32 -p tcp --dport 2222 -j TPROXY --tproxy-mark 0x1/0x1 --on-ip=127.0.0.1 --on-port=37215]
[   2.480]    INFO github.com/netfoundry/ziti-edge/tunnel/intercept/tproxy.(*tProxyInterceptor).intercept: Adding rule iptables -t mangle -A NF-INTERCEPT [-m comment --comment dariusz01 -d 3.3.3.3/32 -p udp --dport 2222 -j TPROXY --tproxy-mark 0x1/0x1 --on-ip=127.0.0.1 --on-port=38385]
[  14.277]    INFO github.com/netfoundry/ziti-edge/tunnel/intercept/tproxy.(*tProxyInterceptor).accept.func1: received connection: 3.3.3.3:2222 --> 3.3.3.3:49230
[  19.658]   ERROR github.com/netfoundry/ziti-sdk-golang/ziti/internal/edge_impl.(*edgeConn).Connect: {connId=[1]} timeout waiting for response
[  19.658]   ERROR github.com/netfoundry/ziti-edge/tunnel.Run: zt.Dial(dariusz01) failed: timeout waiting for response

I don’t see any logs at the edge router corresponding to this request for connection, which makes sense that it comes back with a timeout.

when I do simple curl to port 3022 from client’s loptop, I get a response:

~/sandbox$ curl zedeapptest01.centralus.cloudapp.azure.com:3022
curl: (52) Empty reply from server

log at the edge router:

Feb 13 23:52:03 zedeapptest01 ziti-router: #033[34m[85923.820]#033[39m #033[31m ERROR#033[39m #033[36mgithub.com/netfoundry/ziti-foundation/channel2.(*classicListener).listener [tls:0.0.0.0:3022]#033[39m: error receiving hello (receive error (tls: first record does not look like a TLS handshake))

Curl to https not http is probably the issue?

The Duplicate Entry error was fixed in the last push.

1 Like

@dariuszSki Do you have a CLI script that you’re using to set things up? I tried to reproduce the error you were seeing when you had a bind policy, but was unable to reproduce. If you’ve got a script that let’s you see the error, I can try and fix the root cause.

FYI found the issue and put up a PR with the fix here: https://github.com/netfoundry/ziti-edge/pull/84