Rankings are a valuable element for city-comparison purposes since results withdrawn from these comparisons can, eventually, support the evaluation of strategic decisions taken by cities. Smart city rankings are not an exception and, as they draw more attention, the number of them exponentially increases. This paper evaluates the appropriateness of existing smart city rankings for quantifying the materialization degree of the smart city concept. The analysis reveals that current rankings generally overlook indicators of the Information and Communication Technologies dimension. To bridge this gap, this work proposes a methodology based on Multiple-Attribute Decision Making that uses technological criteria for designing smart city rankings. The proposed methodology is evaluated against the cities of New York, Seoul, and Santander. Imbalances between results provided by the studied rankings and our evaluation are detected, which suggests the need for a new insight into more suitable and precise evaluation of the smartness degree of cities.